Tempelis
2020-11-04 16:35:41

@Tempelis has joined the channel

vrabbi
2020-11-04 16:36:01

@vrabbi has joined the channel

jsturtevant
2020-11-04 16:37:28

@jsturtevant has joined the channel

neolit123
2020-11-04 16:40:52

@neolit123 has joined the channel

neolit123
2020-11-04 16:42:05

set the channel topic: Slack channel for the image-builder project: https://github.com/kubernetes-sigs/image-builder

cecile
2020-11-04 16:43:10

@cecile has joined the channel

dongsupark
2020-11-04 16:43:19

@dongsupark has joined the channel

vbatts
2020-11-04 16:43:34

@vbatts has joined the channel

jsturtevant
2020-11-04 16:52:15

I'll kick it off from the thread I started in sig-cluster-lifecycle: could I get a review/approval for the image builder for windows ?  If there is anything holding it back please let me know.
Happy to answer questions or address any feedback.  Thanks!

jsturtevant (https://github.com/jsturtevant)
Assignees
codenrhoden
Labels
cncf-cla: yes, size/XXL
cecile
2020-11-04 17:29:09

lgtm but didn’t approve to get more eyes on it

cecile
2020-11-04 17:30:18

I’m hoping at least @codenrhoden and/or @moshloop / @jdetiber reviews it, and maybe someone who is more familiar with the windows specific scripts than I am like @Kalya Subramanian or @Mark Rossetti

jdetiber
2020-11-04 17:31:57

Will take another look at it this afternoon

moshloop
2020-11-04 17:34:04

It lgtm, but my windows and powershell is rudimentary at best

Kalya Subramanian
2020-11-04 17:34:31

Will take a look today

jsturtevant
2020-11-04 17:45:44

Thanks all!

Mark Rossetti
2020-11-04 23:10:52

I can take a look today

jdetiber
2020-11-04 17:19:51

@jdetiber has joined the channel

moshloop
2020-11-04 17:31:56

@moshloop has joined the channel

Kalya Subramanian
2020-11-04 17:34:23

@Kalya Subramanian has joined the channel

cecile
2020-11-04 21:32:34

heads up I’m seeing the Azure 20.04 build fail in a few PRs with [0;32m vhd-ubuntu-2004: fatal: [default]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: unknown reason"}[0m

cecile
2020-11-04 21:33:12

The timing is strange but I don’t believe this is related to my recent job config update, it looks like a legitimate packer failure

cecile
2020-11-04 21:33:48

nothing has changed AFAIK in the image-builder scripts so this might be an issue with either a new apt package or with a new base image for ubuntu 20.04

cecile
2020-11-04 21:42:48

CecileRobertMichon (https://github.com/CecileRobertMichon)
Labels
cncf-cla: yes
Mark Rossetti
2020-11-04 23:10:44

@Mark Rossetti has joined the channel

codenrhoden
2020-11-05 15:12:06

@codenrhoden has joined the channel

codenrhoden
2020-11-05 15:15:22

yay new channel! @cecile @moshloop just wanted to say I am sorry I still haven’t been active in GitHub doing reviews/development. I’ve still had some ongoing health problems, and I even spent yesterday getting a CT scan and scheduling a follow-up surgery for next Tuesday. 😞 Hopefully after that (plus a bit of healing time) things will return to normal.

cecile
2020-11-05 16:57:56

No problem at all, please take care of yourself! Your health is top priority, everything else can wait. Good luck with the surgery, I hope everything comes back to normal soon for you 🤞

🤞 dongsupark, Maximilian Rink, fabio
:thanks: codenrhoden
kiran keshavamurthy
2020-11-05 15:16:17

@kiran keshavamurthy has joined the channel

Tushar Aggarwal
2020-11-05 15:16:17

@Tushar Aggarwal has joined the channel

Maximilian Rink
2020-11-05 18:45:14

@Maximilian Rink has joined the channel

voor
2020-11-09 17:24:41

@voor has joined the channel

naadir
2020-11-09 17:27:35

@naadir has joined the channel

shysank
2020-11-09 17:29:52

@shysank has joined the channel

Ratnopam
2020-11-10 03:08:01

@Ratnopam has joined the channel

hidekazuna
2020-11-10 06:50:47

@hidekazuna has joined the channel

fabio
2020-11-10 09:33:31

@fabio has joined the channel

nader ziada
2020-11-10 15:31:59

@nader ziada has joined the channel

Lauri Apple
2020-11-12 12:00:51

@Lauri Apple has joined the channel

Andy Townsend
2020-11-16 11:06:54

@Andy Townsend has joined the channel

codenrhoden
2020-11-16 19:30:46

Should we cancel this week’s office hours due to Kubecon NA?

👍 dongsupark
Tushar Aggarwal
2020-11-16 19:36:41

Yes 👍

dongsupark
2020-11-16 20:04:27

Gentle reminder: in the last bi-weekly call, we agreed to merge the Flatcar PR , and address remaining things afterwards incrementally. It would be great to see it merged. 🙂

dongsupark (https://github.com/dongsupark)
Labels
cncf-cla: yes, size/XL
Comments
16
👀 codenrhoden
jsturtevant
2020-11-17 17:25:32

Ready with the Windows PR as well. Thanks to everyone that has reviewed it and tried out out so far!

jsturtevant (https://github.com/jsturtevant)
Assignees
codenrhoden
Labels
cncf-cla: yes, size/XXL
:ack: codenrhoden
Tushar Aggarwal
2020-11-19 18:02:07

Hi folks, I and @moshloop will be giving a talk on "Deep dive: K8s Image builder" at Kubecon'20 for those who are attending.
Time: 2:55 PM EST / 11:55 PDT

kccncna20.sched.com
👍 cecile, dongsupark, naadir, voor
hidekazuna
2020-11-20 00:27:56

Please approve the PR to fix to build ubuntu 20.04 CAPI image for OpenStack.
Simply updated to 20.04.1 and it's checksum since 20.04 ISO image checksum has gone.

prankul88 (https://github.com/prankul88)
Assignees
hidekazuna
Labels
cncf-cla: yes, lgtm, size/XS
David Lai
2020-11-20 23:49:22

@David Lai has joined the channel

Jura
2020-11-21 13:07:24

@Jura has joined the channel

Deepak Sharma
2020-11-23 14:14:13

@Deepak Sharma has joined the channel

Maximilian Rink
2020-11-30 14:42:35

We have found a few bugs with ubuntu and the current http proxy handeling and i've created an PR that fixes them 😄

MaxRink (https://github.com/MaxRink)
Labels
cncf-cla: no, needs-ok-to-test, size/S
Comments
3
Maximilian Rink
2020-12-02 10:47:18

In the quest to make our images build behind an http proxy ive stumbled across another annoyance:
Kubeadm image pull doesnt respect the proxy as its set right now.
Im unsure wht would be the best way to handle this. I guess the easiest way would just be to skip the pulling behind http proxies and then rely on having a working registry for actual bootstrap

voor
2020-12-02 12:24:00

Yes, you should be able to skip image pull with kubeadm, we use a custom role to pull images in a different manner

Maximilian Rink
2020-12-02 12:38:08

and goss also doent play nicely with a proxy used :S

Maximilian Rink
2020-12-02 12:46:16

hmm, its correctly set for my CI, it just seems like packer is not passing it to the GOSS provisioner. Has any of you encountered that behaviour yet?

naadir
2020-12-02 13:27:07

have you set the proxy on containerd ?

naadir
2020-12-02 13:27:37

kubeadm is doing crictl pull, so a proxy setting on kubeadm itself should be ineffective

Maximilian Rink
2020-12-02 13:36:57

no, but i have hoped to avoid that 😄
As i need to modify the systemd service for that

looks like i have to grow my PR once more 😄

naadir
2020-12-02 13:38:12

another way would be to pull the image locally, export it, copy to the machine, import it and then run kubeadm

Maximilian Rink
2020-12-02 13:46:51

yeah, that even more errorprone 😄

Maximilian Rink
2020-12-02 13:47:17

should also do

Maximilian Rink
2020-12-02 13:48:07

that only leaves me with goss to figure out 🤔

Maximilian Rink
2020-12-02 16:07:48

seems like it is not possible to setup the goss provisioner so that the inital download support an proxy
It has env flags, but those arent in play at that stage 😕

The best workaround that i can currently see it to fetch the goss binary in ansible and then set the provisioner to skip the download and to use the prefetched binary

naadir
2020-12-02 16:56:22

eek

Maximilian Rink
2020-12-02 17:02:47

http proxies are ugly tech 😄

Maximilian Rink
2020-12-02 17:02:55

But i dont have a choice

Maximilian Rink
2020-12-02 17:05:16

In the same process ive noticed that packer fetches the ovf directly from the node and not from the vcenter like a lot of other toling does. And it doesnt fall back on the vcenter pull. Have you noticed the same @naadir or have i just a weird packer config running?

naadir
2020-12-02 17:05:54

i was under the impression it functions as a 301 redirect unless you're using a content library?

Maximilian Rink
2020-12-02 17:06:47

pretty sure it proxies it. I can export ovf fine, altough i only have a connection to the vcenter and cant reach any of the hosts directly

naadir
2020-12-02 17:07:08

if you download from the vcenter ui, you'll also get an error unless you accept the esxi tls cert for the host vcenter is telling the browser to download from

naadir
2020-12-02 17:07:30

there could be two different vmomi mechanisms in play....

Maximilian Rink
2020-12-02 17:09:47

hmm, from the browser im getting an URI under cls/data/0e4b042c-45bd-4fea-aea1-38e1a5e2c81c/Ubuntu20.04template-1.vmdk on the vcenter

Maximilian Rink
2020-12-02 17:17:59

at least i have some talkingpoints for tomorrow

naadir
2020-12-02 17:18:51

that looks like a content library to me

Maximilian Rink
2020-12-02 17:20:25

itsnt tho. its a packer-created vm that never got uploaded to a content lib.
And there isnt any configured for the cluster or vcenter anyway

Maximilian Rink
2020-12-02 17:20:41

i guess its vmwareautomagic tho

naadir
2020-12-02 17:20:53

yeah, i'm not an expert on the api

Maximilian Rink
2020-12-02 17:22:25

well, i guess i need to file a bug/feature request with packer to get clarification

💯 naadir
cpanato
2020-11-30 16:54:57

@cpanato has joined the channel

cecile
2020-11-30 17:03:12

Vince Prignano (https://kubernetes.slack.com/team/UCD11GCET)
John Lam
2020-11-30 20:42:26

@John Lam has joined the channel

Sanika Gawhane
2020-12-01 19:32:28

@Sanika Gawhane has joined the channel

Franklin Lee
2020-12-03 23:45:49

@Franklin Lee has joined the channel

cecile
2020-12-04 22:17:18

(will hold until next week)

CecileRobertMichon (https://github.com/CecileRobertMichon)
Assignees
justinsb, akutz, figo, timothysc, detiber, moshloop, codenrhoden, luxas
Labels
approved, cncf-cla: yes, do-not-merge/hold, size/M
voor
2020-12-05 15:36:34

Very exciting and we'll deserved across the board

cecile
2020-12-08 21:03:17

Merged. Going forward, if you are interested in becoming an image-builder reviewer and have been helping with PR reviews/made significant contributions to the project, please open a PR to add yourself to the reviewers list (note that you must already be a member of kubernetes-sigs).

👍 cpanato, dongsupark
🎉 jdetiber, codenrhoden
voor
2020-12-08 21:04:21

Yay! 🎉

dongsupark
2020-12-10 07:50:24

Thanks to Travis for approving the Flatcar PR, but the CI failed , and I cannot say /retest. Can anyone with privilege please give it /ok-to-test?

k8s-ci-robot (https://github.com/k8s-ci-robot)
dongsupark
2020-12-10 21:25:12

Thanks @jsturtevant! Now it is merged. 🙂

codenrhoden
2020-12-10 21:29:28

Woohoo! Thanks for hanging in there @dongsupark and seeing this all the way in. It was a lot of work! Conrats!

🎉 jsturtevant, cecile, dongsupark
dongsupark
2020-12-11 08:22:01

Thank you so much for taking care of it, even during your health recovery. 🙂

TAKAHASHI Shuuji
2020-12-12 11:26:56

@TAKAHASHI Shuuji has joined the channel

Arunkumar Venkataramanan (DeepBrainz AI)
2020-12-12 16:02:55

@Arunkumar Venkataramanan (DeepBrainz AI) has joined the channel

hidekazuna
2020-12-16 10:34:09

@dongsupark I proposed PR Flatcar related. Please review

hidekazuna (https://github.com/hidekazuna)
Labels
cncf-cla: yes, size/M
Comments
5
👍 dongsupark
dongsupark
2020-12-16 10:38:02

Thanks, I will have a look.

Norman Bhaskara
2020-12-16 14:11:59

@Norman Bhaskara has joined the channel

Andy Townsend
2020-12-16 14:34:28

is it possible to launch the images built by image-builder without CAPI? If so, how do you get the kubeconfig?

cecile
2020-12-16 16:16:37

You can build VMs using those images on any infrastructure/cloud provider. Are you asking about building k8s clusters though?

Andy Townsend
2020-12-16 16:22:20

yeah, so I built an AMI (that I eventually plan to add some extra provisioners to install our software via helm charts). I can deploy it via CAPI but I'm wondering if we could also share that AMI with a customer so they could just run it in there account.

cecile
2020-12-16 16:30:35

I’m no AWS expert but I’m sure pretty sure there are ways to share AMIs with other users, @naadir might be able to help

Andy Townsend
2020-12-16 16:32:03

yeah, that part isn't an issue. It's more, does that AMI have to be deployed via CAPI. When I deployed the AMI manually and SSH'd in, it didn't seem to have a valid kubeconfig to be able to run kubectl

naadir
2020-12-16 16:36:47

there's no kubeconfig because kubeadm hasn't been run.

naadir
2020-12-16 16:37:12

if all you want to do is have a single node cluster without cluster api involved, you can run kubeadm init on that machine

👍 cecile
naadir
2020-12-16 16:37:30

that drops /etc/kubernetes/admin.conf at the end of the process

cecile
2020-12-16 16:37:33

The AMI itself is just an OS image with preinstalled components and container images. In order to bootstrap the vm into a Kubernetes cluster you need some mechanism such as CAPI (which uses kubeadm underne) or kubeadm to run

Andy Townsend
2020-12-16 16:38:17

Thanks. I just found the kubeadm init stuff and have just ran that now.

Veekay
2020-12-19 07:37:52

@Veekay has joined the channel

Veekay
2020-12-19 09:55:24

Folks
, any method to estimate how much resource should be planned for image runtime require ment

Mike McGhee
2020-12-19 22:24:53

@Mike McGhee has joined the channel

Kritsada Chinpala
2020-12-28 11:39:51

@Kritsada Chinpala has joined the channel

Zespre Chang
2020-12-28 13:26:03

@Zespre Chang has joined the channel

jsturtevant
2021-01-06 21:28:51

I started work for adding goss validation for windows. It required some updates to the goss provisioner. Is there someone to give a ping for a review?

jsturtevant (https://github.com/jsturtevant)
cecile
2021-01-11 18:02:53

@Tushar Aggarwal maybe?

farodin91
2021-01-07 16:56:17

@farodin91 has joined the channel

Maximilian Rink
2021-01-08 15:59:12

hi folx, any further input for ?

MaxRink (https://github.com/MaxRink)
Labels
cncf-cla: yes, needs-ok-to-test, size/L
Comments
14
Danilo Tiago
2021-01-11 18:25:29

@Danilo Tiago has joined the channel

Veekay
2021-01-13 09:25:40

Hi Folks any deployment architecture for including dive tool

voor
2021-01-13 11:54:24

dive is packaged as a debian and rpm, you can look at the documentation on how to include additional Debian or rpm packages

Maximilian Rink
2021-01-14 10:19:39

Hi Folx, i would like to hear your opinion on this:
We have this internally and i think it might be useful for other folx as well, especially in the dockerhub ratelimit days 🙂

MaxRink (https://github.com/MaxRink)
Labels
cncf-cla: yes, do-not-merge/work-in-progress, needs-ok-to-test, size/M
Comments
2
neolit123
2021-01-14 15:36:03

i can try joining the call today to chat about

randomvariable (https://github.com/randomvariable)
Labels
kind/bug
Comments
8
Lauri Apple
2021-01-14 18:15:36

@Lauri Apple has left the channel

Victor Lu
2021-01-16 11:07:56

@Victor Lu has joined the channel

Carlos Arturo Quiroga
2021-01-24 01:22:42

@Carlos Arturo Quiroga has joined the channel

Hina
2021-01-28 18:42:19

@Hina has joined the channel

JSEvans
2021-02-05 12:05:52

@JSEvans has joined the channel

Maximilian Rink
2021-02-09 19:34:27

@codenrhoden has capv shipped with any ubuntu 20.04 image yet? at least officially

codenrhoden
2021-02-09 19:36:11

Nothing I’ve ever pushed. We really should start doing that. 🙂 Nothing stopping it from happening.

Maximilian Rink
2021-02-09 19:36:44

yeah, but that might be the best point to inbtroduce efi builds as @naadir wanted to do a few months ago 😄

asatish
2021-02-12 17:56:55

@asatish has joined the channel

William Lam
2021-02-17 04:52:55

@William Lam has joined the channel

Maximilian Rink
2021-02-18 19:56:16

hmm, we added ovftool as dependency for vsphere builds, but havent updated the dockerfile yet

Maximilian Rink
2021-02-18 19:59:34

And the download is behind a registration wall, so much for "just adding it to the image"

moshloop
2021-02-19 09:36:22

I have some capacity spinning up soon and plan to tackle this + ova python script in the cli

Maximilian Rink
2021-02-19 09:42:43

the EULA of ovatool makes shipping it in the container imposiible on first glance. Am i right @codenrhoden?

Maximilian Rink
2021-02-19 09:43:50

For now, i've settled for this:

ENV LC_CTYPE=POSIX
ENV OVFTOOL_FILENAME=VMware-ovftool-4.4.1-16812187-lin.x86_64.bundle

ADD $OVFTOOL_FILENAME /tmp/

RUN /bin/sh /tmp/$OVFTOOL_FILENAME --console --required --eulas-agreed && <br> rm -f /tmp/$OVFTOOL_FILENAME
Which requires you to download the bumdle on your own beforehand

Sanika Gawhane
2021-03-26 00:00:04

Hi @Maximilian Rink - did you encounter something like this -

DRoet (https://github.com/DRoet)
Labels
needs-investigation
Comments
21
Maximilian Rink
2021-02-19 17:32:07

And am i the only one that noticed the the OVAs dont have the systemd-timesyncd service enabled?

voor
2021-02-19 20:46:26

Hmm, I see extrarepos and I see extradebs, but I don't see a way to add extra keys. 😞

voor
2021-02-19 21:25:53

Docker build is failing with the following:

#16 132.9 hack/ensure-ovftool.sh
#16 132.9 ovftool must be present to build OVAs. If already installed
#16 132.9 make sure to add it to the PATH env var. If not installed, please
#16 132.9 install latest from .
#16 132.9 make: * [Makefile:90: deps-ova] Error 1
------
executor failed running [/bin/sh -c make deps]: exit code: 2
make:
* [Makefile:592: docker-build] Error 1

John Lafata
2021-03-03 18:55:07

@John Lafata has joined the channel

fabrizio.pandini
2021-03-04 13:35:22

@fabrizio.pandini has joined the channel

fabrizio.pandini
2021-03-04 13:39:03

PSA Changes for cgroup driver are landing in kubeadm and in CAPI; that’s means that now MUST be implemented before shipping 1.21 images.
Please reach out if you see problems in this …

codenrhoden
2021-03-05 16:39:51

Thanks Fabrizio. I still haven’t been able to move past the issues I was showing before. I’m putting the config file in place as expected, but containerd status doesn’t show that the change has been picked up. I probably need to pair with someone early next week to work through it. I know this needs to happen.

fabrizio.pandini
2021-03-05 19:11:11

thank_you

codenrhoden
2021-03-08 16:58:51

No luck on my side with this. same issues with the config change not taking effect. I’d like to pair with @neolit123 when possible to figure it out.

Maximilian Rink
2021-03-05 17:32:28

FYI: we have an containerd CVE
1.4.4 fixes it, so you propably want to bump that before doing a tag @codenrhoden

❤️ Dell R
codenrhoden
2021-03-05 17:34:13

Thanks. Sounds like a good idea

codenrhoden
2021-03-05 17:54:23

containerd bump:

codenrhoden (https://github.com/codenrhoden)
Labels
approved, cncf-cla: yes, size/XS
Comments
1
Jesse Hu
2021-03-07 03:28:41

@Jesse Hu has joined the channel

jsturtevant
2021-03-09 19:05:31

@codenrhoden @cecile

jsturtevant (https://github.com/jsturtevant)
jsturtevant
2021-03-09 19:05:53

it looks like there are Prow flakes happening with regards to networking

codenrhoden
2021-03-09 19:06:46

Thanks for the explanation there… 🤞 yay code freeze!

jsturtevant
2021-03-09 19:08:19

yea, Im not seeing anything in particular.... this one couldn't get ansible:

jsturtevant
2021-03-09 19:08:37

jsturtevant
2021-03-09 19:08:41

is the right link

jsturtevant
2021-03-09 19:09:28

Not sure if there is much we can do on those?

codenrhoden
2021-03-09 19:10:43

doesn’t seem like it

Zakarias
2021-03-10 14:57:20

@Zakarias has joined the channel

Ben Cressey
2021-03-10 18:15:17

@Ben Cressey has joined the channel

jayunit100-pub-chnl-plz
2021-03-11 17:00:54

@jayunit100-pub-chnl-plz has joined the channel

jayunit100-pub-chnl-plz
2021-03-11 17:05:14

<-- hi folks, windows is broke :)

jayunit100 (https://github.com/jayunit100)
codenrhoden
2021-03-11 17:06:50

we’ll have to see if we can find a way to do some sort of Windows CI.

jsturtevant
2021-03-11 17:08:54

we have azure windows ci... is there a way to add OVA?

codenrhoden
2021-03-11 17:52:21

We’ve been working on getting the OVA stuff in place. Still running into basic issues with Packer over a VPN connection between Prow and the cloud provider. Working on it though!

👍 jsturtevant, jayunit100-pub-chnl-plz
Sherif Abdel-Naby
2021-03-13 17:32:44

@Sherif Abdel-Naby has joined the channel

jsturtevant
2021-03-16 20:48:42

@codenrhoden @cecile I opened a pr to mitigate the failures we were seeing:

jsturtevant (https://github.com/jsturtevant)
Labels
cncf-cla: yes, size/S
Comments
1
:thank_you: codenrhoden
asatish
2021-03-17 15:47:18

hi, I am making changes to image-builder to support RHEL8 for vsphere ISO and I can send a PR for it. The VMDK is created and in the step of creating OVF and OVA, images/capi/hack/image-build-ova.py expects to have an entry for rhel8-64 in OSidmap which is not present. I do not know the OSID and version which is needed by vmware. Can someone please help with this info?
OSidmap = {“vmware-photon-64”: {“id”: “36”, “version”: “”, “type”: “vmwarePhoton64Guest”},
“centos7-64": {“id”: “107", “version”: “7", “type”: “centos7-64"},
“rhel7-64”: {“id”: “80”, “version”: “7”, “type”: “rhel764guest”},
“ubuntu-64": {“id”: “94", “version”: “”, “type”: “ubuntu64Guest”},
“Windows2019Server-64”: {“id”: “112”, “version”: “”, “type”: “windows9srv-64”},
“Windows2004Server-64": {“id”: “112", “version”: “”, “type”: “windows9srv-64"}}

for the time being I added this line, script was able to move ahead, but I would like to know the correct id/version/type for rhel8
“rhel8-64”: {“id”: “80”, “version”: “8”, “type”: “rhel8
64guest”},

Sean Smith
2021-03-18 18:28:23

@Sean Smith has joined the channel

Maximilian Rink
2021-03-19 17:42:01

we should propably set for the OVAs, right?

Mateusz Gozdek (invidian)
2021-03-19 18:39:15

@Mateusz Gozdek (invidian) has joined the channel

Mateusz Gozdek (invidian)
2021-03-19 18:45:34

👋 I'm trying to add Flatcar support to image-builder for Azure and I have some questions:

  • Are VHD images required? Or are only SIG images sufficient? I've seen that VHD images are deprecated and actually building Flatcar images out of Marketplace images is not possible due to some limitations around unmanaged disks uage.

  • I'm stuck at the following error message: the Shared Gallery Image to which to publish the managed image version to does not exist in the resource group mat-dev. I can't quite figure out what I'm doing wrong.

jsturtevant
2021-03-19 19:52:52

We use the VHD's for cluster-api-azure for the sample images since sigs are do not allow public images. If there is limitations I believe it would be find to do only sigs but at this we wouldn't be able to add automated tests to cluster-api-azure I think.

jsturtevant
2021-03-19 19:53:45

for #2 did you modify to include new flatcar definition?

jsturtevant
2021-03-19 19:55:16

I believe we just added gen2 support that only went to sigs for a reference:

jsturtevant
2021-03-19 19:55:31

alexeldeib (https://github.com/alexeldeib)
Assignees
mboersma, jsturtevant, CecileRobertMichon
Labels
approved, cncf-cla: yes, lgtm, size/M
Mateusz Gozdek (invidian)
2021-03-19 19:56:58

We use the VHD's for cluster-api-azure for the sample images since sigs are do not allow public images. If there is limitations I believe it would be find to do only sigs but at this we wouldn't be able to add automated tests to cluster-api-azure I think.
Thanks, that make sense. I'll work on making VHD's to work.

Mateusz Gozdek (invidian)
2021-03-19 19:57:16

2 I already figured out, though It's quite tricky with packer to get all parameters right

👍 jsturtevant
bavarianbidi
2021-03-22 13:30:01

@bavarianbidi has joined the channel

jdetiber
2021-03-23 16:32:03

Has anyone else noticed any issues with recent capi image-builder images related to cloud-config and cloud-final services not running on Ubuntu 18.04 with recent builds?

naadir
2021-03-23 16:33:18

not again

jdetiber
2021-03-23 16:33:47

well, I'm not quite sure if it's an issue of my own making with the raw image builder PR 🙂

cecile
2021-03-23 18:35:26

Didn’t run into any issues when we built new k8s images this week for azure

jdetiber
2021-03-23 18:36:35

Yeah, I figured out the issue, since I'm trying to use the ec2 datasource in an unknown environment I needed to lay down a config file in /etc/cloud-ds-identify.cfg so that the systemd generator would do the right thing...

Gurpreet singh
2021-03-26 04:27:00

@Gurpreet singh has joined the channel

Vignesh Goutham
2021-03-26 17:27:06

@Vignesh Goutham has joined the channel

Vignesh Goutham
2021-03-26 17:34:50

Hello guys 👋, I’m trying to do a capi image build for aws ami. It looks like the ansible errors out saying it needs to be root to perform yum commands on the remote ec2-instance. The interesting part is the build works completely fine on mac (local) and also on an ec2-instance running amazon linux 2. It only errors out when run on amazon linux 2 container.
I created this issue as well with logs -
I’d appreciate it, if anyone has ideas to go about debugging this.

vignesh-goutham (https://github.com/vignesh-goutham)
Labels
kind/bug
Gabriel Silva
2021-03-26 19:01:22

@Gabriel Silva has joined the channel

Sean Smith
2021-03-29 21:14:31

Hey I’m curious if others are using Image Builder like I am, or I’m doing this in a “bad” way. Currently I’ve got a repo which has our ansible customizations and has image-builder as a git submodule (pinned to a specific tag), my CI pipeline copies the ansible customization into the ansible directory (since they aren’t open source/public) and i’m using the makefile based on the image builder book.

jdetiber
2021-04-01 14:23:01

Instead of using it as a git suubmodule, could use use the published container image and bind mount in your customizations?

👍 Sean Smith
Sean Smith
2021-04-01 18:09:10

I could, that makes sense, I didn’t realize that was an option, I’ll lookup the images and start using them.

Sean Smith
2021-04-01 18:16:15

ok yeah that will work perfectly, just need to copy around some files in the CI pipeline because I can’t chose where my repo is mounted

Amit Mishra
2021-04-01 14:00:02

@Amit Mishra has joined the channel

jayunit100-pub-chnl-plz
2021-04-01 16:04:17

anyone goin to office hrs

jsturtevant
2021-04-01 16:06:21

it's next week

jsturtevant
2021-04-01 16:06:33

happens every 2wks

jsturtevant
2021-04-01 16:08:53

https://calendar.google.com/calendar/u/0/r/week/2021/4/8?eid=Y3I5dHJwbmVucjVvcWEybXBnO[…]0MDhUMTUwMDAwWiBjYWxlbmRhckBrdWJlcm5ldGVzLmlv&pli=1&sf=true Is a link I think

accounts.google.com
Kubernetes Moderator Service
2021-04-01 17:33:09

@Kubernetes Moderator Service has joined the channel

Kubernetes Moderator Service
2021-04-01 17:33:09

@Kubernetes Moderator Service has joined the channel

jayunit100-pub-chnl-plz
2021-04-06 21:44:14

@codenrhoden any objection to a hack/serve.py script just so we have a canonical example of how to serve artifacts on a url in image-builder?

jayunit100-pub-chnl-plz
2021-04-06 21:47:50

re:

jayunit100 (https://github.com/jayunit100)
Labels
kind/feature
Comments
2
codenrhoden
2021-04-06 21:56:35

I think that could be useful, yeah. If people are really enterprising, it can also be a way to speed up local builds that are purely upstream. I image it ends up just being a wrapper around the python http serve module?

🎉 jayunit100-pub-chnl-plz
jayunit100-pub-chnl-plz
2021-04-06 21:58:01

yeah , exactly, not anything special but something to make bikeshedding less dangerous bc theres a canonical implementation to borrow

jayunit100-pub-chnl-plz
2021-04-06 21:58:43

want to make clear my intentions here :) goal not to solve all problems but just to say "hey, heres an example" so we can get away from abstract descriptions of something that can be done in like 4 lines of code :)

👍 codenrhoden
jayunit100-pub-chnl-plz
2021-04-06 21:50:29

(it basically is a way for anyone downstream to have a minimal impl of an artifact server) so that theres an end to end example upstream of how to use image builder w/ custom inputs

vrabbi
2021-04-08 13:11:13

Is there a way to run the image builder on an existing machine? I have a need to add a baremetal server to a capv cluster and need to prepare the node. Has any done something like this or any guidance on how to extract the relevant parts of image builder in order to prepare the node correctly?

Peri
2021-04-08 16:41:37

@jsturtevant sorry it was yours!! 😄 this has some notes on running ansible on its own

😄 jsturtevant
Peri
2021-04-08 16:41:43

cc @codenrhoden

vrabbi
2021-04-08 16:56:37

Awesome thanks!

jsturtevant
2021-04-08 17:04:56

note that it is windows specif but its the same general idea

Peri
2021-04-08 16:31:39

@Peri has joined the channel

vrabbi
2021-04-12 10:12:40

I see that in the url.yaml ansible task for downloading k8s binaries no retry logic has been added however for other downloads like for k8s images there is retry logic. Is there a reason for this? Would this be possible to add. If this is something that others would find helpful i can create a pr with the suggested change

Peri
2021-04-16 08:55:17

i dont think there is any particular logic about that it is likely just who made the tasks, maybe put an issue in to track this, it seems like a sensible ask to me

👍 vrabbi
vrabbi
2021-04-16 09:11:23

https://github.com/kubernetes-sigs/image-builder/issues/595

vrabbi (https://github.com/vrabbi)
Labels
kind/feature
Peri
2021-04-16 09:11:53

perfect thanks @vrabbi!

Nate
2021-04-28 02:31:13

@Nate has joined the channel

chrisg
2021-04-29 14:31:45

@chrisg has joined the channel

Subhash
2021-04-30 09:04:06

@Subhash has joined the channel

Nate
2021-04-30 21:52:00

For vsphere images, is there a packer config for ova disk size? The templates are creating with a 20Gi disk. I can't find where to increase that.

voor
2021-05-02 00:03:52

The base image is just for the ova, when actual clusters are stood up you can change the cloned images disk size

Nate
2021-05-02 00:08:00

ok, that's what I figured but am not seeing something correctly with the provision. have set disk size in cluster config yaml, but seems not to take. probable user error.

voor
2021-05-02 12:01:57

Where are you setting the value?

Nate
2021-05-03 02:28:45

in the cluster config yam via VSphereMachineTemplate:

549-kind: VSphereMachineTemplate
1578-metadata:
1588- name: dev-cluster
1608- namespace: dev
1625-spec:
1631- template:
1643- spec:
1653- cloneMode: linkedClone
1682- datacenter: natelab
1708- datastore: esxi-local-1
1738: diskGiB: 75

jsturtevant
2021-05-03 17:08:06

@Peri we increase the disk size for Windows I think?

Peri
2021-05-03 18:01:08

Yep we have 80gb I think?

Peri
2021-05-03 18:01:25

There is a way to change it in image builder too

Nate
2021-05-03 22:14:00

that was my initial thought, but couldn't figure out how to set it with image builder. but i am only using the available base and clone images and adding a deb with image builder.

Peri
2021-05-06 15:16:42

perithompson (https://github.com/perithompson)
Assignees
codenrhoden, kkeshavamurthy
Labels
approved, cncf-cla: yes, lgtm, ok-to-test, size/M
:1000: Nate
Nate
2021-05-01 05:50:32

are there any plans to allow a template file to pull in modules/templates? sort of like a kustomization patch pattern.

Nate
2021-05-01 19:25:50

i suppose I was looking at the template purpose from the wrong angle. use template for bare min cni and csi, then use post provisioning stage with kustomize for the rest.

voor
2021-05-02 00:02:54

For things like cni and csi I would look at cluster resource sets or something like kapp-controller

voor
2021-05-02 00:03:07

Those shouldn't be anywhere close to image building

Nate
2021-05-02 00:06:35

ah, yes, that is what I'm using. I didn't post this in the right channel. meant to post to base cluster-api.

Maximilian Rink
2021-05-04 09:33:18

containerd 1.5 is out
How long do we want to stick with 1.4.x?

Maximilian Rink
2021-05-04 09:34:19

registry.config_path
looks promising and should make exposing the mirror settings easier.

Scott Carrier
2021-05-05 10:58:33

@Scott Carrier has joined the channel

Chris Privitere
2021-05-05 16:25:02

@Chris Privitere has joined the channel

Chris Privitere
2021-05-05 16:26:41

Is there a way to disable exports of the ova file with image-builder? Image builder is the preferred way to build images for things like TKG downstream but in that scenario there's actually no need to have a .ova file created. The template on the vsphere cluster that packer creates is sufficient. But i cannot for the ilfe of me figure out a way to overload the export config with a later .json file. Any ideas?

codenrhoden
2021-05-25 17:26:19

This is pretty much a limitation with out we have our Packer config files structured at the moment. But I am in full agreement with you:

codenrhoden (https://github.com/codenrhoden)
outofmemory
2021-05-06 03:43:54

@outofmemory has joined the channel

Loty Gero
2021-05-12 00:07:31

@Loty Gero has joined the channel

J.P. Poveda
2021-05-12 21:55:39

@J.P. Poveda has joined the channel

JorgeArteiro
2021-05-14 11:33:25

@JorgeArteiro has joined the channel

Ulan Manas
2021-05-14 12:48:07

@Ulan Manas has joined the channel

Hoon Jo
2021-05-16 01:27:48

@Hoon Jo has joined the channel

jsturtevant
2021-05-25 17:23:06

fyi on some potential release changes in packer goss provisioner @codenrhoden

caulagi (https://github.com/caulagi)
Comments
4
codenrhoden
2021-05-25 17:24:31

interesting catch there! Thanks for commenting on that.

K8sVoodoo
2021-05-25 18:09:32

@K8sVoodoo has joined the channel

Thomas Jordan
2021-06-04 04:09:12

@Thomas Jordan has joined the channel

Kevin Breit
2021-06-04 15:28:23

@Kevin Breit has joined the channel

Kevin Breit
2021-06-04 15:28:33

I’m trying to build an OVA in image-builder from a CICD pipeline and it’s giving me an error saying python3 isn’t in the path. But here’s output showing that’s not necessarily true.

User's Python3 binary directory must be in $PATH
Location of package is:
Location: /root/.local/lib/python3.8/site-packages
$PATH is currently: /root/.local/lib/python3.8/site-packages:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Apricote
2021-06-08 10:31:47

@Apricote has joined the channel

roehlc
2021-06-11 14:00:27

@roehlc has joined the channel

Maximilian Rink
2021-06-15 11:56:37

hmm, what is the difference between raw and qemu build targets?
as far as i can see its only the disk image format. anything im missing here?

codenrhoden
2021-06-16 21:06:30

The expectation is that they will diverge. We'll see if that actually happens... I'm with you, though.

Maximilian Rink
2021-06-16 22:01:12

yeah, ive come across that building images fore bare metal and as ironic supports cqow ive left it at qemu for now. Saves my builder a lot of space 😄

Maximilian Rink
2021-06-16 22:02:30

in general, im going to put up a pr for both builders tomorrow, bringiong goss and the additional commands from opther builders to this builders as well.

Maximilian Rink
2021-06-16 22:03:12

they worked fine for me and i see no reason to diverge here 😄 correct me iof im wrong @dan @jdetiber

jdetiber
2021-06-17 00:18:58

sgtm

Maximilian Rink
2021-06-17 09:36:51

😄

MaxRink (https://github.com/MaxRink)
Labels
cncf-cla: yes, needs-ok-to-test, size/M
Comments
2
William Lam
2021-06-15 13:46:26

Not sure if this is the right place to ask, but it looks like the documentation for building your own custom OVA images to use with TKG is out of date with latest v1.20.5 at the least the directory structure and instructions no longer match up

docs.vmware.com
vrabbi
2021-06-15 15:39:42

Have you tried with the updated docs for tkg 1.3?

vrabbi
2021-06-15 15:40:07

They completely changed the mechanics and how to do build an image

William Lam
2021-06-15 15:53:16

Ah, I did try subbing in 1.3 but maybe it should have been 1.3.1. Let me take a look at the docs again, shame it doesn’t have a drop down to jump to specific version

William Lam
2021-06-15 15:53:53

ah I see it now, thanks Scott!

vrabbi
2021-06-15 15:56:01

Its all in docker now and is much easier to use

vrabbi
2021-06-15 15:56:21

If you have any issues with it let me know. More than happy to help out

William Lam
2021-06-15 17:03:18

Yes, this is MUCH easier! I’m able to build a default image now

Do you happen to know where the packer definitions are within container image to specify which ISO to use?

vrabbi
2021-06-15 17:38:37

it should be in /home/imagebuilder/packer/ova/

vrabbi
2021-06-15 17:39:00

the files there are:
centos-7.json
esx.json
linux
OWNERS
packer-common.json
packer-haproxy.json
packer-node.json
packer-windows.json
photon-3.json
rhel-7.json
ubuntu-1804.json
ubuntu-2004.json
vmx.json
vsphere.json
windows
windows-2004.json
windows-2019.json

vrabbi
2021-06-15 17:39:17

which OS are you trying to change the image for?

codenrhoden
2021-06-16 21:07:56

@William Lam All the JSON files in the container image are just copied straight from the repo, so it has all the same default whether you are using a container or not. All of those variables can be overriden via flags or your additional custom JSON file so you don't have to go in and edit files within the container.

William Lam
2021-06-16 21:08:27

Yup, figured it out with some pointers from @vrabbi

:partyk8s: codenrhoden
M.C. Attaway Jr
2021-06-15 23:07:44

@M.C. Attaway Jr has joined the channel

whites11
2021-06-16 14:56:43

@whites11 has joined the channel

whites11
2021-06-16 14:58:40

Hello all 👋
I might be missing a point, can anybody rubberduck me with my attempts to build an azure SIG image?
I deployed a VM on azure, installed all the deps and I am trying to run make build-azure-sig-ubuntu-1804
I am getting this error

Error initializing core: error interpolating default value for 'crictl_url': template: root:1:66: executing "root" at crictl&#95;version...>: error calling user: test</pre>But the crictl_version var is defined in one of the packer files:<br><pre>$ grep "crictl&#95;version" packer/azure/**.json<br>packer/azure/packer.json:    "crictl&#95;url": "<a href='https://github.com/kubernetes-sigs/cri-tools/releases/download/v{{user'></a>crictl_version}}/crictl-v{{usercrictl_version`}}-linux-amd64.tar.gz",
packer/azure/packer.json: "crictl_version": "1.21.0",
What am I missing? Thanks

codenrhoden
2021-06-16 21:10:00

@cecile @jsturtevant I know it's last minute, but I'm not going to be able to make office hours this week (tomorrow morning). Attendance has been pretty low, but I'm hesitant to just cancel it. My plan is just to write in the doc that I won't be there, but am hoping any discussion makes its way into notes. Does that seem reasonable?

cecile
2021-06-16 21:11:47

let’s see if anyone else has agenda items by EOD and if not we can just cancel

cecile
2021-06-16 21:12:05

I don’t have the host key btw, does anyone else have it?

codenrhoden
2021-06-16 22:08:57

I do have it, yeah. 🙂 That's something I could pass along to you

jsturtevant
2021-06-17 15:52:04

Doesn't look like we have an agenda. Should we cancel for today then?

👍 cecile
Maximilian Rink
2021-06-17 13:52:45

btw, what are your thought around systemd-boot? together with systemd248+ it is possible to get working discencryption with TPM autounlock that isnt a hassle to maintain. Im currently looking into that for our bare metal nodes

naadir
2021-06-17 13:54:56

we'd want official distro support i think

naadir
2021-06-17 13:55:25

using it where the distro doesn't support it might be problematic in terms of support from the vendor

Maximilian Rink
2021-06-17 14:24:49

arch it is then 🙂 😛

naadir
2021-06-17 14:26:00

yolo

Maximilian Rink
2021-06-17 14:26:56

but tbh, systemd-boot has prooven more stable in our env than grub2 and configuration is sooo much easier

naadir
2021-06-17 14:43:25

need to convince the debian folk, not me

naadir
2021-06-17 14:44:05

i went to Lennart's talk on systemd-boot a few years ago at Fosdem and was mostly convinced then

naadir
2021-06-17 15:07:09

that said, @Patrick Daigle, would it be possible to ask Canonical if they would support Ubuntu images which use systemd-boot as the bootloader? Getting full disk encryption for free would be a bonus

Patrick Daigle
2021-06-17 15:09:09

No promises, but I can try bringing this up in our ongoing conversations and touch points.

:thank_you: naadir, Maximilian Rink
Patrick Daigle
2021-06-17 15:07:50

@Patrick Daigle has joined the channel

Maximilian Rink
2021-06-21 15:30:29

hey folks, if im not missreading this we are assuming that open-vm-tools is a base package in our goss checks


and

that is messing up my bare metal images, that obviously dont have that tooling.
is there any other provider requireing open-vm-tools besides vmware?
If not im ammending my PR for qemu goss tests to remove that from defaults

Maximilian Rink
2021-06-23 21:37:48

so, ive opened up a few PRs for the QEMU and RAW builder @jdetiber @codenrhoden


jayunit100-pub-chnl-plz
2021-06-22 18:50:56

@cecile or @codenrhoden can you reopen ? Feel like its
better for someone who knows the image-builder stuff to quickly make that PR and merge as needed.

I dont mind doing it but, its an OWNERS file so probably is something that can be done w/ a quick handshake by the regular imagebuilder owners :)

jayunit100 (https://github.com/jayunit100)
Labels
cncf-cla: yes, size/XS
Comments
4
codenrhoden
2021-06-22 19:59:56

yeah I'm happy to do it

🎉 jayunit100-pub-chnl-plz
codenrhoden
2021-06-23 16:37:32

Opened

codenrhoden (https://github.com/codenrhoden)
Assignees
jsturtevant, jayunit100, CecileRobertMichon, perithompson
Labels
approved, cncf-cla: yes, do-not-merge/hold, size/XS
❤️ jayunit100-pub-chnl-plz
trierra
2021-06-22 23:51:01

@trierra has joined the channel

Kevin Breit
2021-06-25 19:15:48

  {% if kubernetes_semver is version('v1.21.0', '>=') %}
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
{% endif %}
Is there a reason the SystemdCgroup section requires 1.21?

voor
2021-06-26 19:36:44

Just became the default in 1.21 so it was easier than making sure the underlying machine image was configured accordingly

Kevin Breit
2021-06-26 19:54:46

Makes sense. I think we are fine without it under 1.20. We will be on 1.21 within a few months is my guess anyways. Thank you!!!

👍 voor
Maximilian Rink
2021-06-28 18:18:20

yes, but you /really/ need it for 1.21, as otherwise you wont be able to bootstrap a node with the kubeadm settings capi uses for 1.21

trierra
2021-06-25 20:14:28

Hello! I'm might be missing something in the build flow and it would be awesome if someone can point me to the right direction.

I run make quick-release in order to build my changes, but I don't see a kubelet image despite I have modified code there

~/g/s/k/kubernetes (px-translation-library) [1]> ls -la _output/release-images/amd64/
total 422624
drwxr-xr-x 2 oksana oksana 4096 Jun 25 12:00 .
drwxr-xr-x 3 oksana oksana 4096 Jun 25 12:00 ..
-rw------- 2 oksana oksana 126286848 Jun 25 12:00 kube-apiserver.tar
-rw------- 2 oksana oksana 120957952 Jun 25 12:00 kube-controller-manager.tar
-rw------- 2 oksana oksana 133050368 Jun 25 12:00 kube-proxy.tar
-rw------- 2 oksana oksana 52460544 Jun 25 12:00 kube-scheduler.tar
oksana@dev-onaumov ~/g/s/k/kubernetes (px-translation-library)>
It looks like the kubelet has been built according to logs, but I can't find an output that I could load into a docker container and deploy later
+++ [0625 11:55:37] Building go targets for linux/amd64:
cmd/kube-proxy
cmd/kube-apiserver
cmd/kube-controller-manager
cmd/kubelet
cmd/kubeadm
cmd/kube-scheduler

Maximilian Rink
2021-06-28 18:19:57

uhm, the image-builder doesnt have quick-release target, have you maybe picked the wrong channel?

trierra
2021-06-28 20:28:05

yeah, looks like that..

Nicklas Sörensen
2021-06-29 10:00:20

@Nicklas Sörensen has joined the channel

Kevin Breit
2021-07-01 03:00:37

How do you go about making sure the images are up to date when in production? I’d assume I’d want to routinely rebuild the images with a unique name (include date most likely) and update my CAPI machines to use them. If so, is there a way to add a suffix to the tmeplate name?

jsturtevant
2021-07-02 18:45:34

Basically yes. In practice, I think it will be slightly different for each provider and they way they are distributed.

jsturtevant
2021-07-02 18:48:22

@jackfrancis has been experimenting with a way to keep nodes fresh in a long running cluster: . It is an interesting approach you might checkout for ideas

Stars
<p>6</p>
Language
<p>Python</p>
Kevin Breit
2021-07-02 18:53:18

Interesting. I’ll need to look into this a little more.

Kevin Breit
2021-07-02 18:53:32

My environment right now is VMware.

voor
2021-07-02 21:00:51

There's a content library for updated images of you're using the built-in kubernetes

Kevin Breit
2021-07-02 22:11:19

With Kamino?

voor
2021-07-03 01:39:42

No with vSphere

Kevin Breit
2021-07-03 02:39:02

I’m using ClusterAPI. Do you know if those images work for CAPI?

Kevin Breit
2021-07-03 02:39:14

I’d assume not through

Bala
2021-07-01 06:45:41

@Bala has joined the channel

Maximilian Rink
2021-07-03 15:40:38

hmm, we are currently not cleaning up the netplan folder after we build images. Is there a specific reason for that or is it just oversight 😄

naadir
2021-07-05 10:55:37

oversight. i also noticed we're not cleaning up the journald log

Maximilian Rink
2021-07-05 15:33:09

hmm, is there an ansible module that can do that? from the man page --vacum** only cleans up archived data from journald

Maximilian Rink
2021-07-05 15:33:45
journalctl --rotate
journalctl --vacuum-time=1s
Maximilian Rink
2021-07-05 15:33:51

might do the trick tho

Maximilian Rink
2021-07-05 15:34:11

still feels dirty to use cmds 😄

naadir
2021-07-05 15:34:32

welcome to use python-dbus if you want

naadir
2021-07-05 15:36:11

though you've not lived until you've run COM inside .NET inside Ruby inside Java to interact with Windows logging.

Maximilian Rink
2021-07-05 15:38:43

uhm, my worst sinn actually is realtime programming in windows 😄
Running a Soft-PLC on an Intel Atom with one core to run a pharmaceutical machine is pain

Maximilian Rink
2021-07-05 15:38:58

for everyone involved

naadir
2021-07-05 15:41:18

😬

naadir
2021-07-05 15:42:02

anyway, Poettering said in 2015

The journal will not be bus enabled any time soon, as dbus-daemon logs to it, and this would hence mean a cyclic dependency where dbus daemon logs to journald, and journald uses dbus-daemon's IPC services... This can be fixed only when we have kdbus where the whole broken idea of userspace IPC is gone...

Maximilian Rink
2021-07-05 15:43:58

😄

MaxRink (https://github.com/MaxRink)
Labels
cncf-cla: yes
cecile
2021-07-07 17:39:34

📣 Hi all, since the last couple of image-builder office hours have had very low attendance and no agenda topics we decided to try something new. We’ll add the agenda entry to the notes a week before the next meeting so folks can add topics throughout the week (just added the one for next week in Image Builder Office Hours - Google Docs) and if there are no topics added by the night before the meeting we’ll cancel the meeting occurrence. Let’s try this out a couple of times and see how it works out.

👍 cpanato, voor, codenrhoden
👀 cpanato
:thanks: codenrhoden
cecile
2021-07-07 17:39:49

cc @codenrhoden @jsturtevant

:ack: jsturtevant
cecile
2021-07-14 22:58:54

no agenda added, should we cancel?

cecile
2021-07-15 00:26:51

Cancelling tomorrow’s meeting due to no agenda items

👍 jsturtevant
Zachary Wachtel
2021-07-09 21:19:48

@Zachary Wachtel has joined the channel

Kaushambi Sharma
2021-07-11 10:15:00

@Kaushambi Sharma has joined the channel

Christopher Dziomba
2021-07-12 10:44:03

@Christopher Dziomba has joined the channel

tamal
2021-07-12 21:29:21

@tamal has joined the channel

lastcoolnameleft
2021-07-13 20:35:51

@lastcoolnameleft has joined the channel

Tanner Watson
2021-07-14 01:58:23

@Tanner Watson has joined the channel

Vignesh Ram S
2021-07-14 10:19:19

@Vignesh Ram S has joined the channel

Hoon Jo
2021-07-15 05:00:02

@Hoon Jo has left the channel

Maximilian Rink
2021-07-19 21:09:48

@codenrhoden @naadir the ova ci is broken as credentials got rolled

codenrhoden
2021-07-19 21:16:47

it's concerning that those could get rotated without me knowing about it... Unfortnately I can't really dig into this right now (and of course it's like 1 week after we got it all turned on and working), so I may have to just turn it off for now.

naadir
2021-07-19 21:18:32

argh, ok

codenrhoden
2021-07-19 22:33:35

I sure wish the /override command worked. 🙂 I can open a PR to make this OVA CI optional later tonight, but have to run out the door right now. kid's tee ball game

cpanato
2021-07-20 09:05:37

will do that

cpanato
2021-07-20 09:09:15

cpanato (https://github.com/cpanato)
Assignees
codenrhoden, CecileRobertMichon
Labels
cncf-cla: yes
Vignesh Goutham
2021-07-19 22:59:25

Hello all, I’m trying to run goss validate on an image built by image-builder with overrides manually, and I seem to be missing something. Packer running goss at end of run seems to work fine, but when I try to run goss validate with all the variables filled in, it seems to fail with index of nil pointer error, which I’m pretty sure is my mistake somewhere. Appreciate any help here. I filled in the goss-vars.yaml as well.

root@vignesh-6wdxg:~/image-builder/images/capi/packer# sudo goss -g goss/goss.yaml --vars goss/goss-vars.yaml --vars-inline '{"ARCH":"amd64","OS":"Ubuntu","PROVIDER":"ova"}' validate
Error: could not read json data in goss/goss-command.yaml: template: test:61:24: executing "test" at : error calling index: index of nil pointer

Vignesh Goutham
2021-07-19 23:16:04

I made it work providing all the overrides through --vars-inline, but would be interesting to know why populating the goss-vars.yaml file won’t work.

Maximilian Rink
2021-07-20 08:08:11

https://github.com/containerd/containerd/releases/tag/v1.5.4
We should probably update contained 😅

Maximilian Rink
2021-07-20 14:47:05

MaxRink (https://github.com/MaxRink)
Labels
cncf-cla: yes, size/XS
👍 bavarianbidi, cecile
Maximilian Rink
2021-07-20 20:25:49

hmm, im also seeing some GOVC import errors on new OVAs created by image builder

Maximilian Rink
2021-07-20 20:26:12
[20-07-21 19:18:39] Uploading ubuntu-2004-2021-07-20T18-58z-kube-v1.20.9-disk1.vmdk... Error: Post "": unexpected EOF
govc: Post "": unexpected EOF
Maximilian Rink
2021-07-20 20:26:32

Manual imports through the UI are fine tho, so i dont know what is going on here

Maximilian Rink
2021-07-20 22:03:07

Hmm, this is so weird, it seems to be a ci-only thing.
from my local machine /everything/ works, altough im literally just tunneling all traffic through the machine that runs the CI jobs

Jayesh
2021-07-23 18:29:19

@Jayesh has joined the channel

cecile
2021-07-26 21:40:01

Hi all, added an agenda entry for this Thursday’s office hours in Image Builder Office Hours - Google Docs - please add your topics items before Wednesday EOD

cecile
2021-07-29 01:21:35

Cancelling tomorrow’s meeting due to no agenda items

:ack: codenrhoden
:thanks: codenrhoden
Kevin Breit
2021-07-27 22:13:33

I’m (probably) going to need to load an internal root certificate into my images for CAPV. Is this possible with Image Builder?

Maximilian Rink
2021-07-29 10:52:42

yes, impliment your role that does that and set the custom role var

Maximilian Rink
2021-07-29 10:54:37

would be great if i could get a few more eyes on and , especiually the latter as i needed to touch goss and ansible for a few providers 🙂

Maximilian Rink
2021-08-12 14:34:07

anyone? :D

Pankaj
2021-08-04 05:44:21

@Pankaj has joined the channel

naadir
2021-08-10 15:04:46

i'm getting lots of DMs with folk wanting . tbh, i think it's a bit suspect but seems like loads of orgs deploy .local domains

randomvariable (https://github.com/randomvariable)
Labels
size/S, cncf-cla: yes
Comments
1
naadir
2021-08-10 15:05:00

cc @codenrhoden

vrabbi
2021-08-12 15:38:47

This would be great! I know .local shouldnt be used but i bump into .local domains all the time and in k8s without this workaround its an absolute nightmare

naadir
2021-08-12 17:01:25

it's merged @vrabbi but you'll need to enable it manually as it's technically a "leak", it's in the docs.

vrabbi
2021-08-12 17:01:40

awesome!

vrabbi
2021-08-12 17:01:42

thanks

vrabbi
2021-08-12 17:02:25

building an image now to test it out in my .local lab (i created a dedicated lab to test out .local domain issues with K8s)

👍:skin_tone_4: naadir
ysung
2021-08-19 16:55:33

@ysung has joined the channel

Aarush Goyal
2021-08-24 17:59:42

@Aarush Goyal has joined the channel

Maximilian Rink
2021-08-25 09:28:56

@codenrhoden @naadir @jdetiber we were thinking about pushing some of our baremetal code for image building upstream

Maximilian Rink
2021-08-25 09:30:05

In addition to the packer based builder we also have debootstrap for installing ubuntu, which is significantly faster and produces slimmer images

✔️ Shailesh Pant
Maximilian Rink
2021-08-25 09:30:21

is there any interest to add that to image builder?

naadir
2021-08-25 10:32:19

pinging @Anusha Hegde , @Jamie Monserrate, @Dharmjit and @Shailesh Pant from our Edge team who might have some opinions

i've been interested in debootstrap and the rhel-derivative equivalent. I would start off with an issue describing what you want to do, and we'll need active maintainers etc...

Shailesh Pant
2021-08-25 10:51:50

Thanks @naadir for adding, This most definitely is interesting and like you mentioned an issue detailing the approach and proposal would be a great place to start collaborating 👍

Shailesh Pant
2021-08-25 10:32:44

@Shailesh Pant has joined the channel

Dharmjit
2021-08-25 12:52:31

@Dharmjit has joined the channel

Anusha Hegde
2021-08-25 12:54:15

@Anusha Hegde has joined the channel

cecile
2021-08-25 18:20:37

Does anyone have agenda items / topics for tomorrow’s office hours? Image Builder Office Hours - Google Docs

codenrhoden
2021-08-26 15:13:29

I'm not seeing anything. Going to put the 'cancelled' banner in there I guess. 🙂

robbie
2021-08-26 12:00:55

@robbie has joined the channel

Vignesh Goutham
2021-08-30 20:02:36

Hello all, does anyone know of a specific reason why we do not use the minimal images for ubuntu and use the full size server images? Are there any caveats if we switch to the minimal ones?

whites11
2021-08-31 16:14:37

Anyone with experience in azure SIG image building for windows? CI for my PR fails and I can't understand what I'm missing 🙏

whites11
2021-08-31 16:15:13

This is last error I'm getting:

Build 'sig-windows-2019' errored after 1 second 43 milliseconds: the Shared Gallery Image to which to publish the managed image version to does not exist in the resource group image-builder-e2e-ey01nu[0m

jsturtevant
2021-08-31 19:45:29

i've started to look into this

:thank_you: whites11
jsturtevant
2021-08-31 19:46:40

I am a bit confused as I don't know why it passed, I thought each image name needed to be unique

jsturtevant
2021-09-01 01:28:41

opened

Mounish S
2021-09-03 13:37:15

@Mounish S has joined the channel

Jamie Monserrate
2021-09-06 04:16:34

@Jamie Monserrate has joined the channel

codenrhoden
2021-09-09 14:46:55

No agenda items in the doc this morning, wrote a note saying today's office hours are canceled. I've got a few things that probably make sense to add for next go around in 2 weeks.

👍 whites11, cecile
yoctozepto (Radosław Piliszek)
2021-09-13 10:26:29

@yoctozepto (Radosław Piliszek) has joined the channel

Vignesh Goutham
2021-09-16 00:46:53

Hello all, Im trying to do a qemu image build of ubuntu 2004 and I’m getting stuck on waiting for ssh to become available Any pointers here would be helpful.

==> qemu: Waiting for SSH to become available...
2021/09/15 23:44:29 packer-builder-qemu plugin: [INFO] Attempting SSH connection to 127.0.0.1:2753...
2021/09/15 23:44:29 packer-builder-qemu plugin: [DEBUG] reconnecting to TCP connection for SSH
2021/09/15 23:44:29 packer-builder-qemu plugin: [DEBUG] handshaking with SSH
2021/09/15 23:45:03 packer-builder-qemu plugin: [DEBUG] SSH handshake err: ssh: handshake failed: read tcp 127.0.0.1:51640->127.0.0.1:2753: read: connection reset by peer

jdetiber
2021-09-17 21:32:34

Ubuntu 18.04 CAPI image builds are currently broken for ova, qemu, and raw images, I created a PR to bump the version of the ISO we are using to fix:

Domingos Rodrigues
2021-09-21 01:21:14

@Domingos Rodrigues has joined the channel

Jeremi Piotrowski
2021-09-23 14:14:52

@Jeremi Piotrowski has joined the channel

Jeremi Piotrowski
2021-09-23 14:16:31

hi, I submitted the following PR to improve Flatcar support in image-builder and would love to get some feedback/reviews ; we're currently trying to bring up vsphere support as well

dongsupark
2021-09-24 15:02:26

Reviewed, thanks!

Jeremi Piotrowski
2021-09-24 15:45:40

thanks, i made adjustments and responded to your comments

Amim Knabben
2021-09-26 01:26:40

@Amim Knabben has joined the channel

farhan
2021-09-28 13:42:48

@farhan has joined the channel

Kevin Breit
2021-09-29 14:51:04

I need to install some trusted CA certificates into images for CAPI. These need to be for image repositories so it goes in directories specific to that repository address. Is there a built in task I can use for this or do I need to create a custom role?

vrabbi
2021-09-29 16:02:23

Why do this inside the image? That seems to me like a bad idea. If you need to replace the certs its a pain. I find it much easier to use files directives in kubeadmconfigtemplate and in kubeadmcontrolplane objects. That also allows for an easy rolling update with a changed cert without needing to change the template

Kevin Breit
2021-09-29 16:07:24

Interesting. Do you have an example of those pieces?

Kevin Breit
2021-09-29 16:11:50

Ahhhh just the content?

Kevin Breit
2021-09-29 16:41:49

What’s the difference between putting it in kubeadmconfigtemplate and kubeadmcontrolplane? Is it one is focused on control plane nodes and the other is worker nodes?

vrabbi
2021-09-29 17:07:18

Yep

vrabbi
2021-09-29 17:07:22

Exactly

vrabbi
2021-09-29 17:08:54

I use YTT for templating but here is an example of how i do it

Kevin Breit
2021-09-29 18:10:39

And is this enough to get containerd to connect to an image repository whose certificate is signed by this CA?

Kevin Breit
2021-09-29 18:14:34

I’m trying to get the magic incarnation to make this work. I had it working last night and can’t reproduce it.

vrabbi
2021-09-29 18:54:15

If the ceet is in the OSes trusted CA certs then containerd should respect that

vrabbi
2021-09-29 18:54:21

At least it works for me

Kevin Breit
2021-09-29 18:54:54

PEM format too, right? Not DER or base64?

vrabbi
2021-09-29 18:55:17

Either base64 or PEM

vrabbi
2021-09-29 18:55:35

Which capi provider are you using?

vrabbi
2021-09-29 18:55:56

Look at the file directive in CAPV for the kubevip static pod

vrabbi
2021-09-29 18:56:12

You can see there how pem encoded would be passed

vrabbi
2021-09-29 18:56:25

It can alsi be b64 if you prefer

vrabbi
2021-09-29 18:56:46

I would not wish DER on my worst enimies so i dont know

Kevin Breit
2021-09-29 18:59:58

CAPV

Kevin Breit
2021-09-29 19:00:26

I’m new to PKI and all this so the format stuff is a bit of black magic. Is there a way to see which format something is encoded?

vrabbi
2021-09-29 19:01:08

Im not to farmiliar with that either but im sure that tools exist

vrabbi
2021-09-29 19:01:13

I just dont know

Kevin Breit
2021-09-29 19:18:22

Why are you installing it in the /etc directory structure then moving it later? Is it support for Proton?

vrabbi
2021-09-29 19:28:40

Photon and Ubuntu so need to support both

Kevin Breit
2021-09-29 19:50:21

I may have it working with your configuration. I’m not sure why it didn’t work when I was doing it manually but it’s possible I “tied a knot” earlier in the process that prevented it from working.

Kevin Breit
2021-09-30 04:21:42

https://community.arubanetworks.com/browse/articles/blogviewer?blogkey=bb199da9-c629-46a4-b810-7dc4b2216e59

community.arubanetworks.com
Kevin Breit
2021-10-11 14:52:18

Are you using image-builder to create your Photon image?

vrabbi
2021-10-11 15:13:27

Yes. Havent built photon for a bit of time but yes

Kevin Breit
2021-10-11 15:29:17

Is the process the same for Ubuntu and Photon? I'm trying Photon and it's not injecting the kickstart file

vrabbi
2021-10-11 17:55:34

Its the same

Kevin Breit
2021-10-12 14:43:34

*Thread Reply:* None

Kevin Breit
2021-10-12 14:44:06

This is what I’m seeing on proton

Kevin Breit
2021-10-12 14:45:53

    - PACKER_VAR_FILES=/opt/image-builder/images/capi/config.json make build-node-ova-vsphere-photon-3
That’s my build line

vrabbi
2021-10-12 14:47:08

im away travelling for 2 weeks and dont have my environment readily available to take a look. if you dont figure it out by then i will try and reproduce and see if i can figure out why its failing for you

Kevin Breit
2021-10-12 14:49:02

Thanks. Enjoy your travels!!!

Kevin Breit
2021-11-05 23:44:53

Not sure if you’re back but I still haven’t figured out the Proton issue.

Amim Knabben
2021-09-29 19:26:48

besides the local builds, is there a rule/best practice to bring some of the targets using docker?

Amim Knabben
2021-09-29 19:27:40

I have been struggling with the make deps-ova setup, in a lot of cases the python/deps gets a mess

jdetiber
2021-09-29 19:32:49

I've seen some instances where our dependency scripts have tried to install things in non-ideal locations before, it doesn't hurt to run it containerized if that is the case with the deps-ova target

✔️ Amim Knabben
Abhishek Mitra
2021-10-01 07:37:57

@Abhishek Mitra has joined the channel

Abhishek Mitra
2021-10-01 07:38:21

can someone help with a basic question . i was able to deploy my own custom AMI with cluster-api image-builder . However that doesnt show up in "clusterawsadm ami list" . Let me know how i can use my own custom AMI while deploying a cluster on AWS using the cluster-api . i have been searching for pointers in the documentation but couldnt find any
am sure this is something basic

voor
2021-10-01 12:02:30

You need to set the AMI id in the AWS machine template

Abhishek Mitra
2021-10-01 16:52:42

thanks for the pointer, i see the spec has the following

spec:
template:
spec:
iamInstanceProfile: nodes.cluster-api-provider-aws.sigs.k8s.io
instanceType: t3.large
sshKeyName: #####

Abhishek Mitra
2021-10-01 16:53:09

so here we need to add "image": "amiid" OR "image":"NAME"

Abhishek Mitra
2021-10-01 16:53:21

like capa-ami-centos-7-1.20.10-00-1631556951

Abhishek Mitra
2021-10-01 20:41:32

nm, figured it out . thanks !

voor
2021-10-02 13:07:51

Apologies, for folks that are finding this and haven't figured it out:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSMachineTemplate
metadata:
...
spec:
template:
spec:
...
ami:
id: ami-012345678910

Apricote
2021-10-04 12:23:07

Regarding the CAPI image: make build-qemu-ubuntu-2004 currently fails for me because no matching host key type found (full error message in a reply).

I think this is related to the latest OpenSSH release 8.8 () which disabled ssh-rsa with sha1 keys.
When I set the options mentionend in the OpenSSH release notes (ANSIBLESSHARGS="-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedAlgorithms=+ssh-rsa" make [...]) it works again.

Should I open a ticket for this? I think we have to options here:

a) Update the Key (images/capi/cloudinit/{idrsa.capi.pub,user-data})
b) Add these options to the default ANSIBLE
SSH_ARGS

openssh.com
Apricote
2021-10-04 12:27:14

Error with context:

==> qemu: Executing Ansible: ansible-playbook -e packer_build_name="qemu" -e packer_builder_type=qemu -e packer_http_addr=10.0.2.2:8482 --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url= containerd_sha256=591e4e087ea2f5007e6c64deb382df58d419b7b6922eab45a1923d843d57615f pause_image=k8s.gcr.io/pause:3.4.1 containerd_additional_settings= containerd_cri_socket=/var/run/containerd/containerd.sock containerd_version=1.5.4 crictl_url= crictl_sha256=44d5f550ef3f41f9b53155906e0229ffdbee4b19452b4df540265e29572b899c crictl_source_type=pkg custom_role= custom_role_names= disable_public_repos=false extra_debs= extra_repos= extra_rpms= http_proxy= https_proxy= kubeadm_template=etc/kubeadm.yml kubernetes_cni_http_source= kubernetes_cni_http_checksum=sha256: kubernetes_http_source= kubernetes_container_registry=k8s.gcr.io kubernetes_rpm_repo= kubernetes_rpm_gpg_key=" " kubernetes_rpm_gpg_check=True kubernetes_deb_repo=" kubernetes-xenial" kubernetes_deb_gpg_key= kubernetes_cni_deb_version=0.8.7-00 kubernetes_cni_rpm_version=0.8.7-0 kubernetes_cni_semver=v0.8.7 kubernetes_cni_source_type=pkg kubernetes_semver=v1.22.2 kubernetes_source_type=pkg kubernetes_load_additional_imgs=false kubernetes_deb_version=1.22.2-00 kubernetes_rpm_version=1.20.9-0 no_proxy= python_path= redhat_epel_rpm= reenable_public_repos=true remove_extra_repos=false systemd_prefix=/usr/lib/systemd sysusr_prefix=/usr sysusrlocal_prefix=/usr/local load_additional_components=false additional_registry_images=false additional_registry_images_list= additional_url_images=false additional_url_images_list= additional_executables=false additional_executables_list= additional_executables_destination_path= --extra-vars ansible_python_interpreter=/usr/bin/python3 -e ansible_ssh_private_key_file=/tmp/ansible-key801906084 -i /tmp/packer-provisioner-ansible4185103635 /home/jt/git/image-builder/images/capi/ansible/node.yml
qemu:
qemu: PLAY [all] *
==> qemu: failed to handshake
qemu:
qemu: TASK [Gathering Facts]

qemu: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Unable to negotiate with 127.0.0.1 port 43119: no matching host key type found. Their offer: ssh-rsa", "unreachable": true}
qemu:
qemu: PLAY RECAP
*
qemu: default : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0
qemu:
==> qemu: Provisioning step had errors: Running the cleanup provisioner, if present...
==> qemu: Deleting output directory...
Build 'qemu' errored after 5 minutes 18 seconds: Error executing Ansible: Non-zero exit status: exit status 4

==> Wait completed after 5 minutes 18 seconds

==> Some builds didn't complete successfully and had errors:
--> qemu: Error executing Ansible: Non-zero exit status: exit status 4

==> Builds finished but no artifacts were created.

Local OpenSSH Version:

$ ssh -V  
OpenSSH_8.8p1, OpenSSL 1.1.1l 24 Aug 2021

Kevin Breit
2021-10-04 14:26:41

@Apricote Is it breaking all builds or just certain configs

Apricote
2021-10-04 14:34:19

I only use the build-qemu-ubuntu-2004 target, so i can't tell if it is affecting other targets. Issue happens with latest master 5f3d1d6998a29ac1de63f0ae914bcb266a242078 for me.

Kevin Breit
2021-10-04 14:40:10

I may run a build today and see

Maximilian Rink
2021-10-04 21:42:00

@codenrhoden the ovas created by the vmware tool arent able to be imported into vcenter 6.5 via govc, only via UI

2021-10-04 22:39:57,899 |  ERROR: [04-10-21 22:39:56] Warning: Line 142: Unable to parse 'flags.vvtdEnabled' for attribute 'key' on element 'Config'.
[04-10-21 22:39:56] Warning: Line 143: Unable to parse 'flags.vbsEnabled' for attribute 'key' on element 'Config'.
govc: file does not exist
I suspect its failing because 6.5 cant make sense of the NVRAM config in the OVA, altough the OVA is still targeting 6.5
Do you know someone to pas that bugreport onto? 😄

kiran keshavamurthy
2021-10-12 18:13:16

What does the vmx-version in the OVF say?

Maximilian Rink
2021-10-12 18:13:31

13

👍 kiran keshavamurthy
Maximilian Rink
2021-10-12 18:15:37

The complete image building happens on vmx13, we basically upgrade the VM after deploy to the max the vcenter supports and set props like uuid for the CSI to function properly via govc

kiran keshavamurthy
2021-10-12 18:18:36

We saw something similar where OVA built using tar had issues getting imported into the vC from the UI. Hence ovftool was introduced as an option to build the OVA. Do you have access to ovftool to try that out? Either way, it seems to be an issue.

Maximilian Rink
2021-10-12 18:19:32

We already use the ovftool

Maximilian Rink
2021-10-12 18:19:47

At least I'm setting the env flag for it

Maximilian Rink
2021-10-12 18:20:00

The UI import is fine actually btw

Maximilian Rink
2021-10-12 18:20:07

That just works

Maximilian Rink
2021-10-12 18:20:19

Only via GOVC seems broken

kiran keshavamurthy
2021-10-12 18:20:29

ah sorry. I misread

richcase
2021-10-07 10:45:46

@richcase has joined the channel

PJ
2021-10-07 19:36:53

@PJ has joined the channel

Nadir Palacios
2021-10-08 02:44:11

@Nadir Palacios has joined the channel

Kevin Breit
2021-10-08 20:41:44

I’m trying to build a Proton image using a setup that already does Ubuntu. But Proton starts and says the network is unreachable and it can’t get the kickstart file.

TheKoguryo
2021-10-12 13:16:06

@TheKoguryo has joined the channel

kiran keshavamurthy
2021-10-12 19:51:19

@Maximilian Rink Regarding the cloud-init bug, Just wanted to check if you are working on an PR based on akutz’s suggestions here

Maximilian Rink
2021-10-12 19:59:11

No, I'm on leave this week

👍 kiran keshavamurthy
Gurpreet Singh
2021-10-14 18:15:09

@Gurpreet Singh has joined the channel

Kevin Latchford
2021-10-16 02:09:00

@Kevin Latchford has joined the channel

Shruthi Rajashekar
2021-10-18 06:05:57

@Shruthi Rajashekar has joined the channel

Shruthi Rajashekar
2021-10-18 06:26:00

Hello,
I am trying to install image builder project in my local environment and I have ansible installed

srajashekar@srajashekar-a01 capi % pip3 show ansible
Name: ansible
Version: 4.7.0
Summary: Radically simple IT automation
Home-page:
Author: Ansible, Inc.
Author-email: info@ansible.com
License: GPLv3+
Location: /Users/srajashekar/Library/Python/3.9/lib/python/site-packages
Requires: ansible-core
Required-by:
srajashekar@srajashekar-a01 capi %
However, when I try to run make deps or make build-do-ubuntu-2004 to build an image, I am facing this error
make deps
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ansible 4.7.0 requires ansible-core<2.12,>=2.11.6, but you have ansible-core 2.11.5 which is incompatible.
Successfully installed ansible-core-2.11.5
User's Python3 binary directory must be in $PATH
Location of package is:
ansible
Location: /Users/srajashekar/Library/Python/3.9/lib/python/site-packages
$PATH is currently: /usr/local/opt/mysql-client/bin:/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/usr/local/go/bin:/Users/srajashekar/bin::/Users/srajashekar/go/bin/ginkgo:/Users/srajashekar/Library/Python/3.9/lib/python/site-packages:/Users/srajashekar/wcp/image-builder/images/capi/.local/bin
make: * [deps-ami] Error 1
make build-do-ubuntu-2004
Requirement already satisfied: pycparser in /usr/local/lib/python3.9/site-packages (from cffi>=1.12->cryptography->ansible-core==2.11.5) (2.20)
User's Python3 binary directory must be in $PATH
Location of package is:
ansible
Location: /Users/srajashekar/Library/Python/3.9/lib/python/site-packages
$PATH is currently: /usr/local/opt/mysql-client/bin:/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/usr/local/go/bin:/Users/srajashekar/bin::/Users/srajashekar/go/bin/ginkgo:/Users/srajashekar/Library/Python/3.9/lib/python/site-packages:/Users/srajashekar/wcp/image-builder/images/capi/.local/bin
make:
* [deps-do] Error 1

voor
2021-10-18 10:32:47

Check out the container image for image builder, it might help with this issue.

Shruthi Rajashekar
2021-10-18 14:31:59

Can you please link me to it?

voor
2021-10-18 15:15:08

https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.9

GitHub
:thank_you: Shruthi Rajashekar
Shruthi Rajashekar
2021-10-18 15:25:28

@voor when I try to run the make target for docker-build I’m getting this error

srajashekar@srajashekar-a01 capi % make docker-build
# We must pre-pull images
docker pull docker/dockerfile:1.1experimental
1.1
experimental: Pulling from docker/dockerfile
612615616619: Pull complete
Digest: sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa6412fff9caa6871649c44
Status: Downloaded newer image for docker/dockerfile:1.1-experimental
docker.io/docker/dockerfile:1.1-experimental
docker pull docker.io/library/ubuntu:focal
focal: Pulling from library/ubuntu
7b1a6ab2e44d: Pull complete
Digest: sha256:626ffe58f6e7566e00254b638eb7e0f3b11d4da9675088f4781a50ae288f3322
Status: Downloaded newer image for ubuntu:focal
docker.io/library/ubuntu:focal
bash: line 1: gcloud: command not found
DOCKER_BUILDKIT=1 docker build --build-arg PASSED_IB_VERSION=v0.1.10-66-gdab2b88f-dirty --build-arg ARCH=amd64 --build-arg BASE_IMAGE=docker.io/library/ubuntu:focal . -t gcr.io//cluster-node-image-builder-amd64:dev
invalid argument "gcr.io//cluster-node-image-builder-amd64:dev" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
make: ** [docker-build] Error 125
srajashekar@srajashekar-a01 capi %

voor
2021-10-18 15:28:21

You don't need to make the container it's already made at the repository url in the release notes

Shruthi Rajashekar
2021-10-18 15:33:08

I see so just pull the container and docker run?

voor
2021-10-18 15:33:52

Yes, you got it

:thank_you: Shruthi Rajashekar
Shruthi Rajashekar
2021-10-18 06:26:21

I installed ansible using pip3 install --user ansible

Shruthi Rajashekar
2021-10-18 06:26:49

I am new to image-builder and ansible and any help to set my env will be appreciated!

codenrhoden
2021-10-21 15:30:48

No Agenda items are present for today's office hours, so I marked the meeting as canceled.

Usama Ahmed
2021-10-22 07:40:03

@Usama Ahmed has joined the channel

Eric Graf
2021-10-29 12:56:15

@Eric Graf has joined the channel

Eric Graf
2021-10-29 13:10:50

Hi 👋, I opened a PR to add support for building OpenStack qemu-kvm CAPI images using a container. I read through the contributor guide but likely still missed something. Please let me know if there is something I need to change. Thanks!

Rohit
2021-11-02 16:21:40

@Rohit has joined the channel

nikparasyr
2021-11-02 16:22:23

@nikparasyr has joined the channel

sedefsavas
2021-11-03 14:05:26

@sedefsavas has joined the channel

sedefsavas
2021-11-03 14:06:35

Hello, will there be a release after cloud-init downgrade fix is in?

jimmi
2021-11-04 11:48:53

@jimmi has joined the channel

jimmi
2021-11-04 11:54:22

i'd love to see (set a default Containerd imports directory) merged and in a release soon. looks like all comments are addressed and the PR is ready to go. @jsturtevant @codenrhoden @Peri sorry to ping you directly but you've been helpful at reviewing the PR so far 🙏

GitHub
👀 codenrhoden
👍 jsturtevant
:git_merge: codenrhoden
codenrhoden
2021-11-04 14:34:09

No agenda items for today's office hours, so I marked it as canceled.

👍 cecile
jimmi
2021-11-04 15:19:46

thanks @codenrhoden @jsturtevant! very much appreciated

❤️ jsturtevant
jimmi
2021-11-04 15:52:14

out of interest, what is the process now for☝️to be used in provider image builds?

jsturtevant
2021-11-04 16:06:22

As in for a release of hte image-builder container?

jimmi
2021-11-04 16:08:17

sorry i'm a n00b in image-builder world... is there a doc or some reference i can read on the process? from what you said, i assume you mean there is a release of image-builder container and then individual providers can use it to build their respective images - is that right?

jsturtevant
2021-11-04 16:22:15

The docs are here: , For capz we usually build our reference images from the main branch but there is also container that has all the components build it in: . Looks like we might be due for a new release there.

jimmi
2021-11-04 16:47:09

thanks @jsturtevant

codenrhoden
2021-11-04 20:39:06

there's actually a 0.1.10 tag and container, I just never drafted the release notes.... But I agree, It's about time for a new one. There's a couple pending PRs that are pretty important, and once those are in I'll tag a new release.

👍 jsturtevant
cecile
2021-11-04 16:33:29

@codenrhoden for what would be the best way to test this? use any of the latest images built from main?

GitHub
codenrhoden
2021-11-04 17:44:35

@cecile that sounds right to me. Any recent images built with the Azure pipelines you have in place would have the latest cloud-init in them, and that cloud-init is buggy i a scenario triggered by CAPV. It doesn't appear to me that CAPZ images are effected, so we are leaving the cloud-init as-is. But yeah, if it turns out that CAPZ images are effected, you would know right away because the images wouldn't join into a cluster

cecile
2021-11-04 17:46:59

ok I can give that a try

cecile
2021-11-04 18:26:15

if tests pass we’re good

GitHub
:thanks: codenrhoden
Maximilian Rink
2021-11-06 17:36:11

@codenrhoden somewhat related: to we have end2end tests for actually standing up a CAPI cluster with the built images from CI on PRs to main?

Maximilian Rink
2021-11-06 17:38:32

If not: we have built something for Metal3 and CAPV internally, but on GitLab CI not on prow. We could port that over tho

codenrhoden
2021-11-08 18:26:27

so far it has been provider dependent. If I understand correctly, CAPZ does does test the images they they build with the Azure pipelines. I think CAPG does too with their nightly builds they've implemented. There is not ongoing E2E tests with up-to-date images for CAPV or CAPA that I am aware of, but I may be unaware. I would have to check with those projects.

Maximilian Rink
2021-11-08 13:21:46

Ubuntu 21.10 and 22.04 🧵

Maximilian Rink
2021-11-08 13:24:58

we need to rearchitect the image builder for newer ubuntu versions somewhat, as they only support subiquity. Ive started a while back with but there are a few more changes that we will need to make for newer ubuntu versions going forward. The question is, do we want to add ubuntu 2110 to image builder to start ironing out bugs before 2204 hits or do we want to wait?

GitHub
codenrhoden
2021-11-08 18:24:43

that's a really good question. Maybe one for office hours? I do remember your PR re: subiquity, and I know it's gotten zero attention. 😞 It's definitely been a case of only looking at things for the LTS releases thus far.

I feel like we've mostly stuck to LTS releases thus far, but shaking things out ahead of time would be beneficial. I think the question becomes, do we remove a release, like 21.10, after 22.04 becomes available?

PJ
2021-11-11 17:56:25

Hi All 👋, I was going through the CAPI image builder code and noticed that base image for this dockerfile could use with an image bump (focal -> rolling). Would you all be open to a PR from me that does this bump?

cecile
2021-11-17 23:56:43

PR welcome. Just wondering, why rolling vs. latest? Isn’t “latest” latest LTS?

PJ
2021-11-18 00:22:41

I am okay with "latest" 🙂 I was assuming that, we would want to keep it to a specific release and update it as needed.
IIUC, this release specific tag, makes it possible potentially to create reproducible builds. But moving to "latest" will potentially mean the image tag is mutable and may result in different base image based on when the build was triggered for the same dockerfile.

cecile
2021-11-18 01:23:45

looking at Ubuntu - Official Image | Docker Hub it looked to me like rolling was even less release specific: “The ubuntu:latest tag points to the “latest LTS”, since that’s the version recommended for general use. The ubuntu:rolling tag points to the latest release (regardless of LTS status).”

hub.docker.com
PJ
2021-11-18 01:38:48

oh wow. I learnt something new today. Thank you @cecile. I will use latest in that case

👍 cecile
PJ
2021-11-22 18:45:43

PR created, waiting for jobs to pass:

GitHub
cecile
2021-11-17 23:58:27

Hi all, are there any agenda items for tomorrow’s office hours? I don’t see anything on the meeting notes currently

codenrhoden
2021-11-18 15:17:24

I didn't see any as of this morning, so I just marked it as canceled. Let me know if you want to change that. I've got a couple things brewing that I'll hopefully put on the agenda in two weeks.

👍 cecile, jsturtevant
Ken Hamric
2021-11-22 16:20:59

@Ken Hamric has joined the channel

Shyam P R
2021-11-23 13:57:05

@Shyam P R has joined the channel

Shyam P R
2021-11-23 15:10:51

Team, in the default images that are created using the image builder project , what are the typical firewall/iptable rules in the images? I could not find any reference in the code. In our case, we are having to disable firewall as otherwise worker node to API Server communication, node to node communication, node port services etc do not work.

GitHub
voor
2021-11-23 19:47:42

Hi @Shyam P R most of those rules are not modified from whatever the upstream provided image is, so you can either modify them prior to image builder or afterwards.

Shyam P R
2021-11-24 03:03:32

Thanks @voor

Daniel Lipovetsky
2021-11-23 18:08:05

@Daniel Lipovetsky has joined the channel

Daniel Lipovetsky
2021-11-23 18:10:13

We recently merged to make it easier to override containerd configuration. However, it turns out that containerd only merges configuration at the section level (). Just wanted to raise awareness of this.

GitHub
GitHub
Francisco Navarro
2021-11-30 21:40:30

@Francisco Navarro has joined the channel

Amim Knabben
2021-12-01 12:44:35

is there a new 0.11 planned to be released?

Alejandro Cortina
2021-12-01 13:28:05

@Alejandro Cortina has joined the channel

mkumatag
2021-12-01 13:50:35

@mkumatag has joined the channel

mkumatag
2021-12-01 13:50:58

can someone help me reviewing this pr - ?

GitHub
:ack: codenrhoden
mkumatag
2021-12-06 09:59:21

@codenrhoden fixed all the review comments, ptal latest commit

codenrhoden
2021-12-02 15:11:18

No agenda items for office hours as of last night, marked it as canceled.
I will use that time to review the above PR, and the outstanding one for adding Rocky Linux to QEMU/Raw builders

codenrhoden
2021-12-02 16:51:13

I do think we should tag a v0.11. I think we are in a good spot for it, with some recent fixes going in for AMI+Amazon Linux 2, and some critical ones for OVAs (NTP fixes, Photon AppArmor).

Anyone know of anything pending that looks important? cc/ @Amim Knabben

Amim Knabben
2021-12-02 17:15:27

we have @Peri updates on a Windows nodes timezone issue + the capability to install openssh from a URL source

:ack: codenrhoden
whites11
2021-12-02 18:32:43

@whites11 has left the channel

Shyam P R
2021-12-07 04:30:51

Team, please review the PR to add Oracle Cloud Infrastructure(OCI) support in image-builder. Sorry this is my first PR, so apologies for any mistakes beforehand itself wrt to procedure/code.

GitHub
:ack: codenrhoden
Shyam P R
2021-12-13 16:13:15

Gentle reminder for this @codenrhoden @cecile @naadir

Shyam P R
2021-12-16 04:57:39

Gentle reminder for this @codenrhoden @cecile @naadir @kiran keshavamurthy

codenrhoden
2021-12-16 05:00:04

Ack. I know Kiran was able to take a look yesterday. I should have time tomorrow. Sorry for the delay.

Shyam P R
2021-12-16 05:14:49

np, Thanks @codenrhoden

Shyam P R
2022-01-07 14:57:09

Gentle reminder for this review @codenrhoden @cecile @naadir @kiran keshavamurthy

:ack: codenrhoden
:done_done: codenrhoden
Shyam P R
2022-01-13 14:22:47

Gentle reminder for the review @codenrhoden , the ova tests are still failing though, for all PRs

Shyam P R
2022-01-17 15:40:03

Gentle reminder for review @codenrhoden

Shyam P R
2022-01-20 15:50:33

Gentle reminder for review @codenrhoden

Shyam P R
2022-01-20 16:33:40

Thanks for the review @codenrhoden, @cecile it would be great to get your review also as your original approval was removed by an update

Shyam P R
2022-01-21 13:34:52

Thanks for the review @cecile

Kevin Breit
2021-12-09 21:59:38

Is there out of the box support for RHEL 8

Vasileios Mansolas
2021-12-10 00:14:28

@Vasileios Mansolas has joined the channel

Kevin
2021-12-10 21:21:51

@Kevin has joined the channel

mkumatag
2021-12-12 11:35:27

One more fix for the capi ansible code: , can someone help me reviewing this PR?

GitHub
mkumatag
2021-12-12 11:35:40

cc @codenrhoden ^^

jsturtevant
2021-12-13 23:37:48

@codenrhoden It looks like OVA ci is failing consistently. It doesn't seem to be blocking test. Is this a known issue?

mkumatag
2021-12-14 06:41:00

Failing with this error:

Build 'vsphere' errored after 2 minutes 9 seconds: Post "": dial tcp 54.70.161.229:443: connect: connection timed out
199

200
==> Wait completed after 2 minutes 9 seconds
201

202
==> Some builds didn't complete successfully and had errors:
203
--> vsphere: Post "": dial tcp 54.70.161.229:443: connect: connection timed out

vrabbi
2021-12-14 07:45:44

I think it may have been blocked from internet access like many other vmc vcenters due to log4j cve. Not sure in this case but ive seen that in multiple vmc instances since the cve was announced

jsturtevant
2021-12-14 21:40:04

looks like this was the case

GitHub
codenrhoden
2021-12-16 04:51:06

sorry I didn't respond here, been a busy few days for me. I had Kiran get this fixed up once we got the right networking and firewall settings in place.

jsturtevant
2021-12-16 16:37:23

thanks!

codenrhoden
2021-12-16 04:55:01

I don't see any agenda items, so marked the office hours as canceled. I'll just throw out that I'd lake to tag v0.11 tomorrow. We've been talking about doing it for a while, and i was personally waiting until a recent Photon OVA issue was resolved, and it has been, so it seems like a good time. I know a few people have asked for it, and there's been a good number of Windows related fixes as well.

👍 cpanato, jsturtevant
jsturtevant
2021-12-16 20:47:30

can we include a docs publish? The windows docs aren't in the live page

codenrhoden
2021-12-16 21:21:20

that's... weird. let me check that out. docs are supposed to be published after every merge

codenrhoden
2021-12-16 21:40:10

I'll have to play with it a bit. It looks like it's not configured correctly. It's link to inside of capi.md, but in the published page it's a 404. I think I see why. It would also be nice to just have WIndows show up in the side-bar nav. It's all in the configuration I think, the docs are getting published fine, so it should be an easy fix.

👍 jsturtevant
jsturtevant
2021-12-16 22:25:20

so it is publishing the latest docs but just the page is missing from the nav?

jsturtevant
2021-12-16 22:25:30

let me know if there is anything can help with

codenrhoden
2021-12-17 21:50:30

@jsturtevant PR to get those Windows docs to show up:

GitHub
❤️ jsturtevant
codenrhoden
2021-12-17 21:50:41

you'll be able to see a preview of the docs generated in the tests

codenrhoden
2021-12-17 21:51:25

The "deploy/netlify" job

jsturtevant
2021-12-17 23:23:05

Thanks!

codenrhoden
2021-12-16 21:22:24

tag v0.1.11 is made. And I finally published release notes for v0.1.10. homer-disappear
Will make sure v0.1.11 notes are done today.

:yay: jsturtevant
🎉 Kevin Breit, vrabbi, Apricote
codenrhoden
2021-12-17 04:53:36

GitHub
mkumatag
2021-12-17 06:38:26

Facing issues while make build-qemu-centos-7 , any idea what is happening?

mkumatag
2021-12-17 07:26:51

Adding following entry in the ansible.cfg helped me but its pretty slow

[defaults]
....
timeout = 120

codenrhoden
2021-12-17 22:08:06

I don't see anything obvious from the screen shot. Unfortunately I don't have any experience with the QEMU builder

mkumatag
2021-12-18 01:46:29

n/m, may be because of some nested virtulization, performance is not that great, will try running on a baremetal and check how this goes!

Amim Knabben
2022-01-06 15:10:15

I have one open PR that seems to be failing consistently on a not related job -

packer build -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/kubernetes.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/cni.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/containerd.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/ansible-args.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/goss-args.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/common.json"  -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/config/additional_components.json"  -color=true -var-file="packer/ova/packer-common.json" -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/packer/ova/photon-3.json" -var-file="packer/ova/vsphere.json" -except=esx -except=local -only=vsphere-clone -var-file="/home/prow/go/src/sigs.k8s.io/image-builder/images/capi/ci-photon-3.json"  packer/ova/packer-node.json
vsphere-clone: output will be in this color.
==> vsphere-clone: Cloning VM...
Build 'vsphere-clone' errored after 11 seconds 801 milliseconds: Error finding network: path 'sddc-cgw-network-8' resolves to multiple networks

GitHub
Amim Knabben
2022-01-17 16:09:29

@codenrhoden Travis, can you PTAL again, cleaned up the open issues

GitHub
codenrhoden
2022-01-06 17:34:09

@Amim Knabben (and anyone with PRs), we are definitely aware of the issues with OVA CI. I'm working to address the problem now (it's an infra issue). I'm hoping to have it resolved in the next day or so, but if it continues to be an issue we can consider making that CI test non-blocking. All OVA CI tests are going to fail right now.

Amim Knabben
2022-01-06 17:34:58

thanks Travis, lmk if you need help on it.

jsturtevant
2022-01-06 18:03:13

is there an issue/pr to track the fixes?

codenrhoden
2022-01-06 18:25:22

No not really. I can write an issue up, though, so it can be referenced. Will be a little while before there is a PR. I'll write one now.

codenrhoden
2022-01-06 18:29:36

GitHub
:thanks: jsturtevant
jsturtevant
2022-01-06 18:43:51

I don't think OVA is marked as required right now...

codenrhoden
2022-01-06 18:49:35

wow, you are right. it was for a while, but I completely forgot that it wasn't anymore.

codenrhoden
2022-01-06 18:49:50

GitHub
codenrhoden
2022-01-06 18:50:41

I would have felt real silly if I went to go turn that on and found it was already there. Well, once this gets sorted out, I'll set it back. 😆

👍 jsturtevant
Shyam P R
2022-01-17 15:36:09

@codenrhoden the ova build seems to be failing, can we make it non blocking? sorry if it was already made that

codenrhoden
2022-01-19 18:50:08

They are currently non-blocking. And I will definitely get to this today.

:party_parrot: Amim Knabben
kuja53
2022-01-07 07:47:38

@kuja53 has joined the channel

cecile
2022-01-07 18:09:47

Hi all, I’m going to start a PR to update the OWNERs file and:

  1. make sure various project owner aliases are up to date

  2. remove image-builder approvers and reviewers that have not been active in the project in 1+ year

  3. propose that we add @jsturtevant and @kiran keshavamurthy to image-builder-maintainers


If anyone has interest in becoming a reviewer, please let me know and we can work together to make it happen.

👍 timothysc, figo
❤️ jsturtevant, figo, codenrhoden, kiran keshavamurthy
cecile
2022-01-07 18:09:53

cc @codenrhoden

cecile
2022-01-07 18:10:53

let me know if there are any objections / considerations I should take into account before doing this. Also happy to split 3 from 1 and 2 if we want to approve the PRs separately.

cecile
2022-01-07 18:12:43

cc @akutz @jdetiber @figo @justinsb @luxas @moshloop @timothysc (you are all listed as maintainers currently)

timothysc
2022-01-07 18:13:51

feel free to rotate me out.

👍 cecile
akutz
2022-01-07 18:27:46

Ditto.

akutz
2022-01-07 18:28:26

If there is an emeritus tag, I'll take it since I created the project along with @jdetiber, but I am no longer actively involved.

❤️ cecile
cecile
2022-01-10 22:52:58

PR opened:

GitHub
akutz
2022-01-07 18:12:55

@akutz has joined the channel

figo
2022-01-07 18:12:56

@figo has joined the channel

justinsb
2022-01-07 18:12:56

@justinsb has joined the channel

luxas
2022-01-07 18:12:56

@luxas has joined the channel

timothysc
2022-01-07 18:12:56

@timothysc has joined the channel

Yiyi Zhou
2022-01-11 19:51:52

@Yiyi Zhou has joined the channel

codenrhoden
2022-01-13 15:11:03

No agenda items for today's office hours. I marked it as canceled in the doc.

oguzdag
2022-01-14 11:09:45

@oguzdag has joined the channel

Yiyi Zhou
2022-01-18 21:49:21

hi, I am new to image-builder. This might be a stupid question. I was ruuning make build-node-ova-local-photon-3 and encountered FileNotFoundError: [Errno 2] No such file or directory: 'vmware-vdiskmanager' I googled this error and people saying VMware Fusion would already have vmware-vdiskmanager built in. Anyone has any ideas to fix this? Thank you!

codenrhoden
2022-01-19 18:48:41

Hi @Yiyi Zhou! It's most likely that the folder that contains vmware-vdiskmanager is not in your PATH environment variable. On my Mac, VMware Fusion adds that executable in this folder:

$ which vmware-vdiskmanager
/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager

codenrhoden
2022-01-19 18:49:04

so you probably need to add /Applications/VMware Fusion.app/Contents/Library to your PATH

Yiyi Zhou
2022-01-19 19:40:05

Thank you Travis! You are right. I added export PATH=$PATH:/Applications/VMware\ Fusion.app/Contents/Library and now it worked.

Yiyi Zhou
2022-01-19 20:22:19

@codenrhoden I have another question. So I didn't change any files and ran make build-node-ova-local-ubuntu-2004 . I opened the vmx file in Fusion, expecting to login with default username(builder) and password(builder) in packer-common.json. But it didn't work.

Yiyi Zhou
2022-01-19 20:50:19

I tried appending the key to the ~/.ssh/authorized_keys file locally, still unable to login.

GitHub
codenrhoden
2022-01-20 17:19:38

Hi Yiyi. Since these images are intended for use with CAPI, they are setup to expect an interaction that CAPI automatically performs. Namely, this is the injection of cloud-init metadata to create a user and add an SSH key.

The best way to do about this when working with Fusion is to run the hack/image-post-create-config.sh script. The way I do it is to import the OVA into Fusion, but do not start the VM. Before you start the VM, run the above script, which will create the capv user and add the SSH key that is present in the image-builder repo. Then start the VM and you can SSH with something like ssh -i cloudinit/id_rsa.capi capv@IP.

I just took a look at the docs, and I'm noticing that is out of date. 😞 It definitely needs to be corrected

akutz
2022-01-20 17:24:20

FWIW Travis the issue was her local copy of the key lacked the correct file permissions, and SSH was balking at her.

akutz
2022-01-20 17:24:44

I pointed her to the hack/image-ssh.sh script, which should perhaps automatically apply the correct perms to the key in the examples area.

codenrhoden
2022-01-20 17:25:10

FYI, the builder user is definitely locked and can't be used. If you are booting the VMX directly (instead of importing the OVA first), you may be able to SSH into it by grabbing the IP from fusion, using the capv user, and pointing to the SSH key found in the cloudinit user, but I haven't done it that way in a while. But that set of files is injected with the metadata automatically here:

codenrhoden
2022-01-20 17:25:14

thanks @akutz

akutz
2022-01-20 17:26:07

And yeah, the builder user gets locked as part of the shutdown command.

GitHub
Yiyi Zhou
2022-01-20 21:32:25

Thank you @codenrhoden @akutz

mkumatag
2022-01-24 11:36:23

@naadir can you please help me reviewing this PR? , I have tested the changes and works fine, cc @codenrhoden

mkumatag
2022-02-07 17:02:15

@naadir @codenrhoden can we merge if no other comments?

mkumatag
2022-02-15 11:29:40

Ping @naadir @codenrhoden

naadir
2022-02-15 14:45:55

I am not working on upstream at the moment. @codenrhoden @Sanika Gawhane, can we get this sorted please?

mkumatag
2022-02-17 10:19:50

@codenrhoden @Sanika Gawhane let me know if any meeting required to discuss this?

jsturtevant
2022-01-25 17:11:10

There is a new issue with WS2022 build in the Azure VHD job. It is an OS packaging issue and I don't have a workaround as of now so will open a PR shortly to disable the job until it is resolved.

jsturtevant
2022-01-25 17:14:49

fyi @codenrhoden @kiran keshavamurthy

:ack: kiran keshavamurthy
codenrhoden
2022-01-25 17:15:29

Thanks @jsturtevant

codenrhoden
2022-01-25 19:47:12

@jsturtevant If you like, me or @kiran keshavamurthy could open the PR to set required: false just to unblock things.

👀 kiran keshavamurthy
jsturtevant
2022-01-25 19:57:00

sorry was in sig meetings but I don't think we should disable all the tests just ws 2022:

GitHub
👍 kiran keshavamurthy
codenrhoden
2022-01-25 19:57:45

of course. picard_facepalm thanks!

jsturtevant
2022-01-25 20:41:20

Tests pass, ready for a review

codenrhoden
2022-01-26 19:28:42

OVA CI is working again. Finally!

❤️ jsturtevant, vrabbi, Amim Knabben, Vignesh Goutham
🙌 Shyam P R
:thank_you: cecile, vrabbi, Amim Knabben
codenrhoden
2022-01-27 15:01:16

No agenda items for this morning's office hours - marked it as canceled

neolit123
2022-02-01 11:51:06

📣 📣 📣 hello, a couple of message on behalf of the SIG Cluster Lifecycle leads
cc @cecile @codenrhoden (from the OWNERS file of image-builder)

  • annual reports: each subproject must nominate a person responsible to fill a short online form (it takes around 5-10 minutes, deadline Feb 11th):

please make sure that the details about your project (owners, contact, meetings) are up to date, if not you can PR changes:

more info on annual reports:


  • KubeCon EU maintainer track: if you are interested in doing a talk, reach out to the SIG leads and they can request a slot for you (deadline Feb 14th)


if you have not subscribed to the SIG mailing list, please do that!


thank you!

:done_ball: codenrhoden
cecile
2022-02-01 17:36:15

@codenrhoden @kiran keshavamurthy @jsturtevant should we jump on a quick call at some point and do the annual report together?

👍 jsturtevant, kiran keshavamurthy
cecile
2022-02-01 17:36:33

And anyone else who wants to participate

codenrhoden
2022-02-01 21:01:32

yeah, that would make sense to me. A lot easier with help!

cecile
2022-02-01 23:06:30

let me know when would be a good time for you

kiran keshavamurthy
2022-02-02 19:11:02

Maybe during the next office hours?

cecile
2022-02-03 00:02:25

I’m going to be oof end of next week, could we do this tomorrow or Friday?

jsturtevant
2022-02-03 16:15:45

today is pretty busy for me but Friday works

👍 cecile
codenrhoden
2022-02-04 16:49:49

I'm pretty free today as well. My group has been trying out "no meeting Fridays", so that works!

cecile
2022-02-04 18:12:51

I’m free too, no meeting Fridays FTW party_parrot

cecile
2022-02-04 18:13:07

wanna do this in 20 minutes?

codenrhoden
2022-02-04 18:31:41

ah, just saw this. I can do it! I didn't prepare though. 😬 @cecile

cecile
2022-02-04 18:32:20

no worries I did it for capz and capi already it’s pretty straightforward we can go through it together

cecile
2022-02-04 18:33:25

@jsturtevant @kiran keshavamurthy are y’all available?

👍 kiran keshavamurthy
jsturtevant
2022-02-04 18:35:22

i am

cecile
2022-02-04 18:37:25

filling out survey for SIG annual report :

Shyam P R
2022-02-02 15:14:43

Team, is there a way to skip installing open-vm-tools using image-builder? We are trying to use image builder in Oracle Linux 8 and in Arm architecture, and open-vm-tools is not present which is causing the build to fail.

Chaitanya
2022-02-07 08:44:47

@Chaitanya has joined the channel

Shyam P R
2022-02-09 12:17:15

Team, any reason we are not bumping the kubernetes version here - to something new? Also I dont see in customisation we have defined how to update kubernetes version - , is that something we should try to add?

GitHub
Apricote
2022-02-09 12:41:11

From the book page that you linked:

The version of Kubernetes to install. The default version is kept at n-2.

As v1.23 is released, the default version should be updated to v1.21

Shyam P R
2022-02-09 14:08:07

Thanks @Apricote, looks like thes section si also misleading

See Customization section below for overriding this value
as the customisation section does not explain the kubernetes properties, If no on else picks it up, I will try to create a PR tomorrow

Shyam P R
2022-02-10 03:42:13

Can I please get an ok-to-test for the PR to bump up kubernetes version to n-2 and minor doc fixes?

GitHub
codenrhoden
2022-02-10 15:40:22

No agenda items for today's office hours. Marked it as canceled. Will plan to take a look at the above PRs during that time.

Jake Plimack
2022-02-11 19:18:34

@Jake Plimack has joined the channel

codenrhoden
2022-02-24 15:33:56

Cancelled today's office hours - no agenda items

Mike Bars
2022-02-24 20:21:00

@Mike Bars has joined the channel

jlieb
2022-03-02 15:17:45

@jlieb has joined the channel

Vignesh Goutham
2022-03-02 19:08:46

Hello all, I’ve been using image-builder for a while now. I was exploring the cli for image-builder in the repo and It looks like it’s not under active development. Is this still on the roadmap? I can help contribute here if this is still a direction the community wants to pursue.

codenrhoden
2022-03-10 15:07:26

yeah, it's definitely true that the CLI isn't getting any attention at the moment. And has been for some time. 😞 There is no current agreed-upoon design of how to approach it, and it's been well over a year since it was even discussed.

For me and a team of engineers I work with, we've been in a state of "yeah, we will have resources to work on that in a month or two" for well over a year now. And honestly that is still the case -- something we still think we want to do, but just don't have the bandwidth/resources to look at it.

Vignesh Goutham
2022-03-10 23:32:35

ah, no worries. I will take a look at the repo and see if I can hash out a design when I get sometime. We can talk about it/discuss more in the office hours whenever I get some direction 🙂 Thanks for the info though.

Shyam P R
2022-03-08 04:46:48

Team, while doing make deps in an Ubuntu box, I get the following

Installing collected packages: wheel, pip
WARNING: The script wheel is installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
WARNING: The scripts pip, pip3 and pip3.8 are installed in '/home/ubuntu/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
but the documentation only talks about add the following in PATH
export PATH=$PWD/.bin:$PATH
Looks like both PWD and $HOME/.local/bin has to be on path. Is it OK if I create a PR to add both directories to path?

codenrhoden
2022-03-10 15:04:17

Seems like it would be fine to me. I wonder if this is some vestige of the docs being written when everything was on Python 2.7 and now things are fully moved over to Python 3.x

Shyam P R
2022-03-10 15:15:35

Thanks @codenrhoden I will create a PR for this

Shyam P R
2022-03-08 12:18:24

Can I please get an ok to test for the PR please

GitHub
👍 jsturtevant
Shyam P R
2022-03-10 01:52:03

Team, please review the above PR

:done_done: codenrhoden
Shyam P R
2022-03-10 15:15:05

@codenrhoden I see you have done lgtm label but not approved for this, any reason? Sorry for bugging you

codenrhoden
2022-03-10 15:15:57

I try not to be the only person who reviews a PR

Shyam P R
2022-03-10 15:17:01

ooh ok, Thanks 🙂

codenrhoden
2022-03-10 15:17:29

sometimes it works out that way anyways, but I at least like to give some time for someone else to look

Shyam P R
2022-03-10 15:18:39

yeah that is great, more eyes the better, I am new to this 🙂, hence wanted to check.

Shyam P R
2022-03-11 11:33:35

Team, can we get another set of eyes on this PR please, we have go the review from @codenrhoden

jsturtevant
2022-03-11 18:22:50

i took a look, lgtm. on minor nit on comment

Shyam P R
2022-03-11 18:35:25

Thanks @jsturtevant, can I get an approve if thats ok from you or @codenrhoden, I will add the comment in the next Pr I have to raise to fix the thread -

Shyam P R (https://kubernetes.slack.com/team/U01MWTK8D6J)
:done_done: codenrhoden
Shyam P R
2022-03-15 13:24:55

Sorry to bother again for this @codenrhoden @jsturtevant but I had to rebase the PR for the ova ci job fix, updated the missing comment as well. Please review

Vignesh Goutham
2022-03-09 19:43:12

Hello team, does anyone have/know some sort of hardening that you use on top of the image-builder image? I found some generic OS hardening like dev-sec repo on GitHub, but I’m curious if something is standard across the k8s community.

codenrhoden
2022-03-10 15:02:48

I don't know of anything standard at the moment. I know of some vendors that use their own additional Ansible roles for hardening, but I don't know what the status of those becoming public is.

Vignesh Goutham
2022-03-10 23:30:41

got it, thanks. Do you mind sharing some links if you have them handy? If not, its fine 🙂

codenrhoden
2022-03-11 19:28:50

Everything I am aware of is non-public right now. I have good reason to believe that may change within a few months, but who knows when it comes to deadlines and releases. They always move around. 🙂

codenrhoden
2022-03-10 15:08:42

There were no items in the office-hours agenda, so I marked it as canceled for today.

Sayali Kulkarni
2022-03-11 07:58:35

@Sayali Kulkarni has joined the channel

mweibel
2022-03-11 09:06:36

@mweibel has joined the channel

Shyam P R
2022-03-14 08:05:04

Team, can someone with the vsphere image builder job knowledge check why the latest builds of pull-ova-all are failing(eg: ), it is failing for all the PRs, so don't think it is PR specific. The error is

[0;32m    vsphere-clone: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to create temporary directory.In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p \"echo /tmp/.ansible\"&& mkdir \"echo /tmp/.ansible/ansible-tmp-1647239786.000533-1078-173893619533102\" && echo ansible-tmp-1647239786.000533-1078-173893619533102=\"echo /tmp/.ansible/ansible-tmp-1647239786.000533-1078-173893619533102\" ), exited with result 1", "unreachable": true}[0m
TIA for the help.

Shyam P R
2022-03-15 16:27:49

Thanks @codenrhoden for fixing this

mweibel
2022-03-18 15:24:33

I randomly have errors with building custom windows 2019 AWS AMIs (using image-builder, pre pulling a few docker containers). It sometimes works and sometimes doesn't.
Issue seems to be the sysprep step in the end (executes C:/ProgramData/Amazon/EC2-Windows/Launch/Scripts/SysprepInstance.ps1). I'm looking into debugging this, but maybe anyone knows right away what that could be?

==> windows-2019: Provisioning with powershell script: packer/ami/scripts/sysprep_prerequisites.ps1
windows-2019: Removing default unattend.xml file...
windows-2019:
windows-2019: TaskPath TaskName State
windows-2019: -------- -------- -----
windows-2019: \ Amazon Ec2 Launch - Instance I... Ready
windows-2019: Using cloudbase-init unattend file for sysprep: C:\Program Files\Cloudbase Solutions\Cloudbase-Init\conf\Unattend.xml
==> windows-2019: Provisioning with Powershell...
==> windows-2019: Provisioning with powershell script: /tmp/powershell-provisioner372200148
windows-2019:
windows-2019: C:\Users\Administrator>reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server" /v fDenyTSConnections /t REG_DWORD /d 1 /f
windows-2019: The operation completed successfully.
==> windows-2019: Provisioning step had errors: Running the cleanup provisioner, if present...
==> windows-2019: Terminating the source AWS instance...

jsturtevant
2022-03-18 17:12:27

are there errors elsewhere in the output? I don't see specific

mweibel
2022-03-18 18:51:50

unfortunately not. There's a sysprep troubleshoot doc from AWS which might help, but requires connecting to the machine. Will try that on Monday.

docs.aws.amazon.com
jsturtevant
2022-03-18 21:45:57

troubling shooting sysprep is tough, sounds like you have a way to do it good luck!

mweibel
2022-03-22 09:08:48

thanks 🙂 funny enough I tried it several times yesterday and today and all of them worked. Fingers crossed that this continues 😉

Flask
2022-03-18 15:40:40

@Flask has joined the channel

Danny Brito
2022-03-21 23:12:50

@Danny Brito has joined the channel

dims
2022-03-24 12:46:15

@dims has joined the channel

dims
2022-03-24 12:46:46

@cpanato please see ( cc @codenrhoden and @cecile )

GitHub
👀 cpanato
:ack: codenrhoden
dims
2022-03-24 12:46:53

@Arnaud (he/him) ^

Arnaud (he/him)
2022-03-24 12:48:39

Damn you’re too fast for me. I just opened a branch for it. 😅

:partycat: dims
cpanato
2022-03-24 12:57:00

nice

Arnaud (he/him)
2022-03-24 12:46:56

@Arnaud (he/him) has joined the channel

Batuhan Apaydın (developer-guy)
2022-03-24 13:49:31

@Batuhan Apaydın (developer-guy) has joined the channel

codenrhoden
2022-03-24 14:40:04

I see there are agenda items for this week's office hours. I've been on PTO this week, and won't be able to attend. I will be back online later today, though, so I'll look through notes and check out what was discussed.

👍 jlieb
cecile
2022-03-24 17:13:12

with return to office, the 8am meeting time doesn’t work well for me anymore as it’s right in the middle of my commute so I likely won’t be able to attend most weeks.

codenrhoden
2022-04-07 15:25:11

I think it's time to consider rescheduling this to a time that isn't at 8am for PDT folks. Our attendance is basically nil at this point. Though I am really appreciative of people leaving notes in the doc last week when no maintainers attended.

There are no agenda items this week, so I'm going to mark the meeting as canceled. Though there was an open through about what's up with the CLI somewhere, I'll track that down and respond to it here.

jlieb
2022-03-24 15:12:43

No maintainers attended @codenrhoden 🙂 There are some questions/requests to the maintainers in the notes. Enjoy your PTO!

mweibel
2022-03-29 06:09:21

👋 I have a few open PRs, could anyone review them, please? (mostly windows/AWS related):


👀 jsturtevant
jlieb
2022-03-31 15:19:12

Hi folks! I'm trying to get merged as part of our effort to ensure Flatcar images are aligned with the rest of the supported distros in terms of GOSS validations (right now some Flatcar builds are either broken due to GOSS errors or unvalidated due to "special" GOSS configuration which diverges from the generic one). This is a small PR (around 40 LOC changed) which ensures Flatcar has a similar GOSS config to the rest of the distros. The PR is blocking other work I'm doing in both image-builder and CAPV (because we need working vSphere images to test CAPV changes which we currently don't have).

@jsturtevant @moshloop looks like you've been automatically tagged as reviewers. Is there anything I can do to make it easier for you folks to review the PR? Alternatively, who could I tag for a review if both of you are busy these days? Thanks! 🙏

Xavier Avrillier
2022-04-07 13:44:52

@Xavier Avrillier has joined the channel

Xavier Avrillier
2022-04-07 13:45:12

@Xavier Avrillier has left the channel

codenrhoden
2022-04-07 15:30:29

@jlieb To answer your question about the CLI and it not compiling (which you fixed, thanks!), it is not currently in use. There were some plans around using it to do a few things, mainly as a pre-processor to generate the Packer configs (which would allow us to have much better logic ((and tests!)) around what the Packer configs look like. There are currently no design docs around it, and the work has no volunteers. The individual who was taking it up moved on to other responsibilities. So as you say, it hadn't been touched in quite some time.

I've been saying for at least a year that I thought I had some team members that may be able to pick up this work "in a couple months", but I've been saying that every few months now and it hasn't happened. Priorities... I wouldn't expect any effort on it in the short term, and if the effort does get picked up again, I think it would be starting from scratch to first get some agreement on goals and direction. I don't think the project has agreement on that, just a few personal opinions.

Vignesh Goutham
2022-04-07 20:25:59

I have also said few weeks back that I’ll take a look and propose some design, but didnt get time. I’m planning on writing something down next week, Hopefully I can get some thoughts and feedback from the community. If others have an idea or design, I’ll love to jump in and contribute as well.

jlieb
2022-04-08 13:55:01

Thanks a lot for clarifying Travis, and thanks for picking this up Vignesh! Just to clarify, there is nothing urgent on my end around this CLI - I was motivated to poke around and fix the build simply because my editor (VS Code) kept complaining about build failures while I was editing JSON files 😆 But of course it's nice to keep enhancing the project.

jlieb
2022-04-08 13:56:25

Is there a difference between the QEMU and the raw build targets in image-builder? I'm a bit confused by the existence of both since both targets seem to be using the qemu Packer builder type.

codenrhoden
2022-04-08 16:27:05



I'm not entirely sure if I'd say this has been the case or not (that the two have diverged. I don't really think they have from the Packer perspective, more from the Ansible perspective (there's a raw and a virt builder_type.

👍 jlieb
Ash
2022-04-11 23:15:39

@Ash has joined the channel

aniruddha
2022-04-12 16:51:26

@aniruddha has joined the channel

Dennis Lerch
2022-04-13 11:42:17

@Dennis Lerch has joined the channel

Amim Knabben
2022-04-19 14:16:01

@codenrhoden Hello, can you help review this one

GitHub
codenrhoden
2022-04-19 15:26:29

Took a look and /lgtm'd. I can approve if needed, too. Thanks for pinging me, as I had missed that folks were looking for my input.

Amim Knabben
2022-04-19 15:31:50

thanks Travis, would be great if you approve, we are in need of it to continue a few downstream testing

:done_done: codenrhoden
:ty: Amim Knabben
Abhay Krishna Arunachalam
2022-04-20 09:40:21

@Abhay Krishna Arunachalam has joined the channel

mkumatag
2022-04-20 15:56:05

Can someone help me reviewing this code - ? cc @codenrhoden

GitHub
mkumatag
2022-04-29 18:18:06

@codenrhoden can you review or suggest someone who can review this PR?

codenrhoden
2022-05-05 15:17:17

Yeah, let me poke some people about it. I had thought this was related to Windows (my mind saw PowerShell), so was expecting windows reviewers to chime in. I was mistaken there.

👍 mkumatag
mkumatag
2022-05-05 15:24:59

n/m, this one is related to a different cloud platform for the ppc64le architecture.

mkumatag
2022-05-10 15:51:43

@codenrhoden wondering if you got a chance to poke anyone to review this PR?

codenrhoden
2022-05-10 15:53:17

Ack. Sorry, I have been swamped and distracted by sick kids lately. @kiran keshavamurthy, if you have any bandwidth, can you take a look at this one? It's on my plate as well.

mkumatag
2022-05-10 16:05:30

hope your kids are doing well now!

codenrhoden
2022-05-10 17:00:01

thanks. one of them is home from school with me, so lots of constant distractions!

mkumatag
2022-05-10 17:08:58

Ah.. happy parenting.. 🙂

mkumatag
2022-05-24 08:09:12

Gentle reminder - @kiran keshavamurthy I have fixed the review comment, can you please review the PR now?

👀 kiran keshavamurthy
mkumatag
2022-05-27 02:05:36

@kiran keshavamurthy ci is green now ptal.. @codenrhoden see if you can also ack..

mkumatag
2022-05-28 02:19:40

@codenrhoden I see Kiran lgtm'ed it, can you ptal the PR as well?

Abhay Krishna Arunachalam
2022-04-20 18:44:24

Hello all, I have a doubt: I'm curious why the open-vm-tools package installation is not a step in the Centos 8 OVA kickstart config. This causes Centos/RHEL 8 OVA builds to be stuck in Waiting for IP state, which then times out after 30 minutes. The VMWare console also shows that Open VMWare tools is not installed in this virtual machine, which corroborates the absence of the installation step. The installation is there in the Centos 7 ks.cfg. Is there a way to get around this?

Abhay Krishna Arunachalam
2022-04-20 18:56:57

cc @kiran keshavamurthy @codenrhoden

kiran keshavamurthy
2022-04-20 19:54:09

I think the centos-8/RHEL-8 support was added for qemu builders only and not OVA. So it has not been validated. We’d be happy to review a PR to add RHEL-8/Centos-8 support for OVAs.

Abhay Krishna Arunachalam
2022-04-20 20:02:09

That is correct. In our CI workflow, we added the following patch for it that adds the rhel-8.json to the OVA packer configs, and it uses the centos/http/8/ks.cfg
https://github.com/aws/eks-anywhere-build-tooling/blob/main/projects/kubernetes-si[…]ge-builder/patches/0009-Add-support-for-RHEL-8-OVA-builds.patch

Abhay Krishna Arunachalam
2022-04-20 20:04:24

could you let me know if this looks right?

Abhay Krishna Arunachalam
2022-04-20 21:36:23

GitHub
Abhay Krishna Arunachalam
2022-04-20 21:37:31



☝️ this issue states

Without open-vm-tools installed, if you try to build centos/rhel OVA the process will remain stuck in "Waiting for IP address"

This is what we're seeing

Pearl
2022-04-21 09:46:48

@Pearl has joined the channel

Abhay Krishna Arunachalam
2022-04-22 21:00:33

When trying to build the Centos/RHEL 8 OVA, I get the error Kickstart file /run/install/ks.cfg is missing. I have tried serving the kickstart config checked in to the repo through both httpdirectory and floppydirs options with appropriate bootmediapath, but the build is not able to find the ks cfg file.

I also get the error Module floppy not found in directory /lib/modules/ . Is this because Centos 8 doesn;t support floppy modules or drivers?

Thilo Fromm
2022-04-26 15:52:27

@Thilo Fromm has joined the channel

Abhay Krishna Arunachalam
2022-04-26 19:34:31

@kiran keshavamurthy Can you take a look at this PR adding support for RHEL 8 OVA builds, including CI?

GitHub
:ack: kiran keshavamurthy
Abhay Krishna Arunachalam
2022-04-27 18:12:20

Addressed comments here. Changed the structure of goss-vars a little to accommodate for different packages for different versions of the same OS. Could you take another look @kiran keshavamurthy

kiran keshavamurthy
2022-04-27 18:15:52

Thanks Abhay. I’ll try to look at it today or tomorrow.

Abhay Krishna Arunachalam
2022-04-27 18:17:51

Awesome, thanks, really appreciate it thanks2

Abhinav Sinha
2022-04-27 06:20:53

@Abhinav Sinha has joined the channel

Abhinav Sinha
2022-04-27 06:35:06

Hi folks! Can someone please confirm if there’s a way to set environment variables for, let’s say, directly setting the values of fields mentioned in images/capi/packer/config/kubernetes.json to build the images instead of using PACKERVARFILES? Or if there’s a workaround to achieve the same instead of having to manually edit the config file every time an image has to be built for a new k8s release?
Any help or clues would be really appreciated, thanks!

jsturtevant
2022-04-27 15:38:21

if you want to change just a few variables you can do something like PACKERFLAGS="-var=KUBERNETESVERSION=1.22.8 -var=ANOTHER_VAR=value" make build-target

:ty: Abhinav Sinha
Abhinav Sinha
2022-05-03 20:11:06

Thanks a lot @jsturtevant! That did the job 🎉

Amim Knabben
2022-05-02 19:00:27

Hey folks, got this issue on a MacOSX when building an OVA image, this happened on the windows-restart provisioner.

Has someone look into a similar issue using image-builder + macosx?

jsturtevant
2022-05-02 19:39:12

I haven't tried from mac os

codenrhoden
2022-05-05 15:16:00

And I haven't tried building Windows, so not sure about this one either. 😕

Kevin Breit
2022-05-05 18:46:19

I have a pipeline which builds a CAPI vSphere image using image-builder and it seems to be erroring but not giving details. Is there a way to look at why it’s happening or get logs from somewhere?

==> vsphere: Typing boot command...
==> vsphere: error typing a boot command (code, down) 28, false: ServerFaultCode: A general system error occurred: Invalid fault
==> vsphere: trying key input again
==> vsphere: Error running boot command: error typing a boot command (code, down) 28, false: ServerFaultCode: A general system error occurred: Invalid fault
==> vsphere: Clear boot order...
==> vsphere: Power off VM...
==> vsphere: Deleting Floppy image ...
==> vsphere: Destroying VM...
Build 'vsphere' errored after 48 seconds 532 milliseconds: Error running boot command: error typing a boot command (code, down) 28, false: ServerFaultCode: A general system error occurred: Invalid fault

jsturtevant
2022-05-06 18:50:45

looks related to packer, you can try turning on more verbose debugging:

packer.io
Kevin Breit
2022-05-06 19:03:40

I was able to get it working locally and not in the pipeline, which I thought was odd. But it works.

Amim Knabben
2022-05-10 18:07:26

hey folks, what is the safer way to add custom Ansible roles in the image-builder? Want to run a few tasks that are not in-tree

jsturtevant
2022-05-10 19:07:04

should be able to use customrole customrole_names?

:ty: Amim Knabben
Amim Knabben
2022-05-10 19:11:38

yea, I'm using volume mount from the host with custom ansible rules, just wondering other options

Amim Knabben
2022-05-11 14:20:34

- simple instructions in the customization

GitHub
jsturtevant
2022-05-11 16:45:14

awesome thanks!

Abhinav Sinha
2022-05-10 19:17:37

Hey folks! I’d like to know if there’s a way to customise the "ami_name" property in the packer config using the packer flag.

Yann Lev
2022-05-12 15:43:51

@Yann Lev has joined the channel

voor
2022-05-16 19:50:48

That feel when image-builder hangs waiting for the instance to come up, but the debug SSH key that it outputs works and connects to a running instance. feelsbadman

codenrhoden
2022-05-19 15:36:42

No agenda items for office hours today, so I marked it as canceled.
As a heads up, I plan to tag the repo today. There's a few things pending that will add some new capabilities, and it would be good to tag right before.

👍 jsturtevant, cecile
jlieb
2022-05-24 17:42:10

Hi folks! Would love to get some eyes on since there are multiple pending PRs blocked on it.
TL;DR: This PR commits the Ignition files used by Flatcar builds to the image-builder repo so that they don't have to be consumed from an external repo.

Thanks!

GitHub
Anirudh gopal
2022-05-24 22:16:52

@Anirudh gopal has joined the channel

Fredrik Björkman
2022-05-25 10:34:18

@Fredrik Björkman has joined the channel

Thomas Illingworth
2022-06-01 14:04:23

@Thomas Illingworth has joined the channel

codenrhoden
2022-06-02 15:45:59

no agenda items today, marking office hours as canceled (I was going to say I am not able to make it anways)
I know there are several open PRs that I've been pinged on. I'm blocking off some time this afternoon to play catch up on reviews.

tanisha banik
2022-06-06 19:38:20

@tanisha banik has joined the channel

Sriraman Srinivasan
2022-06-10 14:07:32

@Sriraman Srinivasan has joined the channel

aniruddha
2022-06-13 09:42:24

make build-gce-ubuntu-1804 giving me this error after I try to build an ubuntu 18.04 image

Jeremi Piotrowski
2022-06-13 09:50:58

Hi aniruddha, are you using a very recent ssh client version? Check this comment and apply the same fix to gce's ANSIBLESSHARGS value and see if that helps

GitHub
aniruddha
2022-06-13 10:32:45

@Jeremi Piotrowski Thanks 😄 it worked:handwithindexandmiddlefingerscrossed:

Marcus Noble (k8s@marcusnoble.co.uk)
2022-06-13 13:42:07

@Marcus Noble has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2022-06-13 13:43:22

Cross-post - (More details in thread)

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)
Jeremi Piotrowski
2022-06-13 13:57:52

This could be the exact same thing as just one message above yours (assuming CAPG -> GCP)

Jeremi Piotrowski (https://kubernetes.slack.com/team/U02E2PT10Q4)
Marcus Noble (k8s@marcusnoble.co.uk)
2022-06-13 14:04:51

I'm not sure it is. I did look at that but that doesn't seem to have any issues with SSH timing out (nor am I using the qemu variants)

Jeremi Piotrowski
2022-06-13 14:30:45

my bad, i thought i saw this resulting in ssh timeouts in the past as well (after multiple retries). it doesn't only affect qemu variants

kuja53
2022-06-13 14:45:25

I guess that best way is to connect to vnc endpoint which packer spawns on every run of image-builder. You could see password and port inside shell output.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-06-13 15:50:10

So, it looks like it was something related to my local network (not sure what exactly) but once I was on our corp VPN is worked first time. 🤷

Shyam P R
2022-06-14 05:41:05

Can I get please get couple of sponsors to join kubernetes-sigs org? I have contributed to image-builder and also planning to move our CAPI Provider repo for OCI to kubernetes-sigs org.

Shyam P R
2022-06-14 05:41:20

PRs




GitHub
GitHub
GitHub
cecile
2022-06-16 20:12:26

Happy to sponsor you

Shyam P R
2022-06-17 05:11:54

Thanks @cecile, i have got 2 sponsors, will ping you in case I need further help

Maximilian Rink
2022-06-20 17:20:11

hmm, ive ran into strange issues with ubuntu and qemu :S
Seems like the audit service isnt installed by default anymore on qemu systems. If i use the same iso and kickstart on an vCenter i still get the package

Maximilian Rink
2022-06-20 17:31:31

qemu: TASK [node : Ensure auditd is running and comes on at reboot] **
241 qemu: fatal: [default]: FAILED! => {"changed": false, "msg": "Could not find the requested service auditd: host"}

Maximilian Rink
2022-06-20 18:22:52

looks like a regression from https://github.com/kubernetes-sigs/image-builder/commit/bc309118a3fe9db9c9d053e8d72d9fad7c43f1fa#diff-d74534cba8de8668a56[…]225ef8dae206955fffbf3135
as that also applies to the raw builder

GitHub
subhasmita
2022-06-25 06:12:53

@subhasmita has joined the channel

jlieb
2022-06-28 15:31:38

Hi folks. should be good to merge now. Testing is easy: make build-qemu-flatcar
I'd love to get this merged as soon as we can since there is a bunch of pending CAPI work that's blocked on this PR.
Thanks! 🙏

GitHub
MeghanaJangi
2022-06-29 11:03:41

@MeghanaJangi has joined the channel

Mohsen Vakilian
2022-06-30 19:58:40

@Mohsen Vakilian has joined the channel

Michal Mazurek
2022-07-02 14:58:58

@Michal Mazurek has joined the channel

Jeff Wu
2022-07-03 05:13:17

@Jeff Wu has joined the channel

jcepeda
2022-07-04 09:47:21

@jcepeda has joined the channel

swan
2022-07-05 10:17:25

@swan has joined the channel

swan
2022-07-05 10:19:11

Hi folks, I am trying to build photon OVA for vSphere, and trying to add a custom OVF property to the OVA, but defining the property in the json and setting that json file as env variable OVFCUSTOMPROPERTIES is not appending the desired property to OVF, could you please correct me if I am doing something wrong?

swan
2022-07-05 10:19:46

I am using target build-node-ova-vsphere-photon-3

swan
2022-07-05 15:37:05

I wanted to this property to be added to Cluster API Provider(CAPI) category

codenrhoden
2022-07-07 22:17:23

@kiran keshavamurthy should know how this works.

kiran keshavamurthy
2022-07-07 22:21:35

@swan are you also setting IB_OVFTOOL=1 to make sure to use ovftool to build the OVA.

swan
2022-07-08 10:03:30

Hey i have got this resolved, thanks @kiran keshavamurthy

Ness
2022-07-08 11:17:47

@Ness has joined the channel

faiq-archived
2022-07-12 17:56:29

@faiq-archived has joined the channel

Deepak Muley
2022-07-13 00:48:43

@Deepak Muley has joined the channel

Ankit Srivastava
2022-07-13 20:55:37

@Ankit Srivastava has joined the channel

Lucas Nascimento da Silva
2022-07-13 20:57:45

@Lucas Nascimento da Silva has joined the channel

opatrick
2022-07-19 03:37:56

@opatrick has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2022-07-19 10:26:49

Does anyone happen to know the minimum required IAM permissions needed by image-builder when building gcp GCP VM images? The docs say to give editor role but this is such large scope of unneeded permissions.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-07-19 12:57:03

I think it's just the following that's needed but would be great if anyone else can confirm.

compute.disks.create
compute.disks.delete
compute.disks.useReadOnly
compute.globalOperations.get
compute.images.create
compute.images.get
compute.images.getFromFamily
compute.images.getIamPolicy
compute.images.list
compute.instances.create
compute.instances.delete
compute.instances.get
compute.instances.getSerialPortOutput
compute.instances.setLabels
compute.instances.setMetadata
compute.instances.setServiceAccount
compute.machineTypes.get
compute.subnetworks.use
compute.subnetworks.useExternalIp
compute.zoneOperations.get
compute.zones.get
iam.serviceAccounts.actAs

Marcus Noble (k8s@marcusnoble.co.uk)
2022-07-20 15:59:44

If anyone is interested, I managed to confirm these permissions are enough today 🙂

👍 Ankit Srivastava, kiran keshavamurthy
Ondrej Kuchar
2022-07-19 17:10:31

@Ondrej Kuchar has joined the channel

Ankit Srivastava
2022-07-20 22:00:59

Building the image build-qemu-centos-7 takes around ~30 mins. Does anyone have any thoughts on how to make it faster? Is there a way we can have incremental image layers so that creating/testing images can be faster?

kiran keshavamurthy
2022-07-27 22:02:40

For vSphere OVAs, we build base images and re-use it (clone builder). This decreases the build times. Not sure if something similar exists for qemu

kopiczko
2022-07-21 10:17:15

@kopiczko has joined the channel

kopiczko
2022-07-21 10:19:20

is ready to merge for a while and solves issues with building images locally with newer OpenSSH versions. I'm not sure who I can assign for that. Can someone please have a look?

GitHub
Jeremi Piotrowski
2022-07-25 10:37:00

hey @kopiczko, i've dropped a comment which i think would allow the CI to pass and allow this PR to get merged

mboersma
2022-08-01 22:08:25

@mboersma has joined the channel

Abhinay Singh
2022-08-09 09:36:50

@Abhinay Singh has joined the channel

Joe Kratzat
2022-08-09 15:51:57

@Joe Kratzat has joined the channel

mweibel
2022-08-12 09:12:26

my PR fails with FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/pip3.7' - anyone know why this could happen?

GitHub
jsturtevant
2022-08-12 19:24:20

that is a expected as its trying to isntall PIP I think, The vhds build and the 2019 image is failing with:

 Builds finished but no artifacts were created.
panic: runtime error: invalid memory address or nil pointer dereference
2022/08/12 07:50:00 packer-builder-azure-arm plugin: [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x1a933ce]
2022/08/12 07:50:00 packer-builder-azure-arm plugin:
2022/08/12 07:50:00 packer-builder-azure-arm plugin: goroutine 226 [running]:
2022/08/12 07:50:00 packer-builder-azure-arm plugin: github.com/Azure/go-autorest/autorest/azure.(Future).WaitForCompletionRef(0xc0001f8cc0, 0x546d9c0, 0xc0006ba000, 0x53f6420, 0xc0000b98b0, 0x540e140, 0xc0001f4a50, 0xc000f08120, 0xc000cd2140, 0xdf8475800, ...)
2022/08/12 07:50:00 packer-builder-azure-arm plugin: /home/circleci/project/packer/vendor/github.com/Azure/go-autorest/autorest/azure/async.go:174 +0x54e
2022/08/12 07:50:00 packer-builder-azure-arm plugin: github.com/hashicorp/packer/builder/azure/arm.(
StepCaptureImage).captureImageFromVM(0xc000562e80, 0x546d9c0, 0xc0006ba000, 0xc0006e7b18, 0x18, 0xc0001a3120, 0x1c, 0xc0005626c0, 0xc000e0e3c0, 0x29)

jsturtevant
2022-08-12 19:24:48

click on "artifacts" on

jsturtevant
2022-08-12 19:25:27

at the bottom of the build log you can see:

jsturtevant
2022-08-12 19:25:28
sig-windows-2019: FAILED. See logs in the artifacts folder.
sig-centos-7-gen2: SUCCESS
sig-ubuntu-2004-gen2: SUCCESS
sig-windows-2022-containerd: SUCCESS
sig-centos-7: SUCCESS
sig-flatcar: SUCCESS
sig-ubuntu-1804: SUCCESS
sig-flatcar-gen2: SUCCESS
sig-ubuntu-1804-gen2: SUCCESS
sig-windows-2019-containerd: SUCCESS
sig-ubuntu-2004: SUCCESS
jsturtevant
2022-08-12 19:26:19

then you can naviagte to artifcats->azure-sigs and open sig-windows-2019.log

mweibel
2022-08-15 15:39:12

ah, thanks for the steps to debug - I didn't see it until now (and figured it out meanwhile myself) 😅

wondering what I could do about it - sounds like an issue with the azure packer plugin. Not sure if a retry would work?

jsturtevant
2022-08-15 17:15:05

looks like an issue in the scripts not naming the image properly. retrying the test for now is ok, looks like we got an issue open to look into it more

steve
2022-08-17 01:47:10

@steve has joined the channel

libsysguy
2022-08-18 16:41:45

@libsysguy has joined the channel

Neeraj Anand
2022-08-20 09:59:44

@Neeraj Anand has joined the channel

Abubakari Sadic
2022-08-21 20:49:37

@Abubakari Sadic has joined the channel

rahav jv
2022-08-23 10:54:27

@rahav jv has joined the channel

rahav jv
2022-08-23 10:54:48

Folks, wanted to check if there are plans to support photon 4.0

Marcus Noble (k8s@marcusnoble.co.uk)
2022-08-24 07:23:49

Morning y'all 👋 Looks like image-builder fails to build the new v1.25.0 release due to the change of default image registry (fails to pull coredns, likely more).
I've opened a PR to change the default registry to the new host:

GitHub
swan
2022-08-24 08:55:48

I faced the same issue, we might need a release for the image-builder with this fix.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-08-24 09:00:43

Yeah. That'd be awesome.
I think it can be worked around by providing that registry value in an override vars file but having it work "out of the box" would be nice 🙂

swan
2022-08-24 09:02:25

oh right, i missed to see that override vars can accept registry key.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-08-24 09:03:24

I haven't tried it though. But pretty sure all those vars can be overridden with a provided --vars-file

swan
2022-08-24 09:03:45

I will give it a try

Marcus Noble (k8s@marcusnoble.co.uk)
2022-08-24 09:03:59

👍 Thanks 🙂

swan
2022-08-24 09:19:28

worked for me, the pull image error is gone, thanks @Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)
2022-08-24 09:19:47

Nice! I'll add that as a note to the PR too 🙂

mimmus
2022-08-24 09:02:24

@mimmus has joined the channel

tariq
2022-08-26 21:11:52

@tariq has joined the channel

Shawn Wang
2022-08-28 07:22:11

@Shawn Wang has joined the channel

voor
2022-08-29 21:27:11

Is there a way to just skip the containerd steps in image builder?

voor
2022-08-29 21:28:06

I guess you could just comment out the

    - include_role:
name: containerd
in node.yml

Amit Sahastrabuddhe
2022-08-30 06:10:29

@Amit Sahastrabuddhe has joined the channel

srm09
2022-08-30 20:13:35

@srm09 has joined the channel

srm09
2022-08-30 20:14:40

Hey folks, can someone chime in on whether the upstream CI is building 1.24 OVAs right now?

Clement Matundu
2022-08-31 19:21:08

@Clement Matundu has joined the channel

Phani Mvs
2022-09-01 06:33:42

@Phani Mvs has joined the channel

Kubernetes Moderator Service
2022-09-07 21:44:16

@Kubernetes Moderator Service has joined the channel

Kubernetes Moderator Service
2022-09-07 22:04:20

@Kubernetes Moderator Service has joined the channel

Sriraman Srinivasan
2022-09-08 13:37:39

I have create an issue for failing image builds (while installing iptables-persistent package). Also have submited PR for fixing the same.

GitHub
:thank_you: kiran keshavamurthy
Sriraman Srinivasan
2022-09-08 13:38:48

Just noticed, @jsturtevant has accepted for test... Thank you

jsturtevant
2022-09-08 16:26:43

Thanks for looking into it!

Cristiano Colangelo
2022-09-08 13:46:26

@Cristiano Colangelo has joined the channel

Vignesh Goutham
2022-09-12 19:13:08

Hi all, I’ve been observing a rather weird issue with vSphere image-builds lately in my environment. If the builder vm path I provide points a sub directory that is more than 1 level deep, rather than a top level directory - ex. capv/vignesh vs capv in the vsphere.json, the builder vm doesn’t seem to accept/take in the preseed cfg/kickstart file properly and loads into the GUI install prompt. Has anyone observed the same? Or any idea if there’s any folder level setting that I can take a look at? My vpshere is 7.0 and Im running the latest commit from image-builder.

Yannam C Chiranjeevi
2022-09-13 13:24:03

@Yannam C Chiranjeevi has joined the channel

Diego Braga
2022-09-14 16:34:43

@Diego Braga has joined the channel

Mitchel Haring
2022-09-15 23:33:04

@Mitchel Haring has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-16 06:16:57

The latest Kubernetes releases - 1.22.14, 1.25.1 - include a version of kubelet that now expects kubernetes-cni@v1.1.1 .
image-builder has it pinned to 0.8.7 causing (Ubuntu) builds to fail without overriding the kubernetescni* vars.
Is there a reason why we pin the version of kubernetes-cni package? Can we remove the version and instead rely on the dependency tree to install the needed version?
The vars could remain for those that need to pin to specific version but the default version can be set to *
to allow ansible to install whichever version suits.

👀 swan
swan
2022-09-16 16:11:54

i am facing same issue, I even tried overriding parameters, but that didn’t work.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-16 21:55:14

I was able to get it to work by adding "kubernetescnidebversion": "**" to my PACKERVAR_FILES json to get it to build. Was that what you tried? Did that not work?

swan
2022-09-19 07:24:12

shall we set all the vars for kubernetescni** ?

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-19 11:49:08

For my test focusing on just ubuntu I only needed to overwrite the one var. ideally all should need updated to match the latest version available I guess.

swan
2022-09-19 12:26:14

I will give it a try, but eventually this has to be handled in image builder itself, isn’t it?

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-19 12:27:12

Yeah 100%. I'm just not sure how best to handle it as it could be a breaking change for some.

swan
2022-09-19 12:30:13

Would it help if we make a new release only with this change?

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-19 12:37:14

I would think that's enough, yes. Though I wasn't able to find out the backwards compatibility for the latest CNI. Not sure how many Kubernetes versions bank would still work as expected.

👍 swan
swan
2022-09-19 13:26:30

Also, I tried setting all CNI vars and it worked for Ubuntu, Flatcar and amazon OS, but for centOS its giving below error.

 Prevalidating AMI Name: capa-ami-centos-7-1.22.14-00-1663588776
==> centos-7: No AMI was found matching filters: {
==> centos-7: Filters: [
==> centos-7: {
==> centos-7: Name: "root-device-type",
==> centos-7: Values: ["ebs"]
==> centos-7: },
==> centos-7: {
==> centos-7: Name: "virtualization-type",
==> centos-7: Values: ["hvm"]
==> centos-7: },
==> centos-7: {
==> centos-7: Name: "architecture",
==> centos-7: Values: ["x86_64"]
==> centos-7: },
==> centos-7: {
==> centos-7: Name: "name",
==> centos-7: Values: ["CentOS Linux 7 x86_64 HVM EBS ENA**"]
==> centos-7: }
==> centos-7: ],
==> centos-7: Owners: ["410186602215"]
==> centos-7: }
Anything changed for centOS too?

swan
2022-09-19 17:10:05

cc @sedefsavas

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-19 18:54:55

We don't work with CentOS images so couldn't say. 😞 Only have experience building Ubuntu and Flatcar.
Are you able to find that image if you search manually in the AWS console?

swan
2022-09-20 07:59:44

no that image is not there only for centOS and the recently released kubernetes versions

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-20 08:29:06

 Prevalidating AMI Name: capa-ami-centos-7-1.22.14-00-1663588776
Are you trying to build that image or make use of it? (Are you responsible for building the CAPA-provided AMIs?)

swan
2022-09-20 11:14:56

Yes you are right, I am responsible to build CAPA provided AMIs

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-20 11:16:05

That makes more sense 🙂
I assume the other Kube versions use the same CentOS base image. Did those also have issues finding the AMI?

swan
2022-09-20 11:16:26

yes other newer k8s releases also have same problem.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-20 11:17:05

Ah, maybe the upstream CentOS AMI name has changed? Let me see if I can find it myself.

:thank_you: swan
Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-20 11:20:08

I can't seem to find anything from the 410186602215 account. Any idea what account that is? Doesn't look like it shares any AMIs

swan
2022-09-20 11:21:28

yeah even I couldnt find it, it is hardcoded in image builder

swan
2022-09-20 11:22:53

oh now I get it, I think you wont be able to find it, coz VMware uses internal AWS account to host these CAPA specific images

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-20 11:25:47

Yeah, looks like the CentOS ownerID to use for public images is 125523088429

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-20 11:26:32

But the names don't match at all.

swan
2022-09-20 11:27:10

I am wondering what might have changed, this worked for last k8s releases 🤔

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-20 11:27:17

If that's an internal VMWare account, wouldn't that mean no one else would be able to build a CentOS based image?

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-20 11:27:42

Maybe something in that account has changed? AMIs incorrectly deleted / set to private?

swan
2022-09-20 11:32:49

i checked the private AMIs, but didnt find there as well.

swan
2022-09-20 11:34:08

If it is account specific I will do some more digging, thanks for the help @Marcus Noble 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-20 11:34:46

No worries 🙂 If I get time I'll try and add CentOS to our pipeline to see if I can get it working or not but sounds like it might be an issue elsewhere.

🙇‍♀️ swan
swan
2022-09-29 16:23:20

I have found the issue for owner id mismatch, could someone take a look at this issue on priority as image building is blocked in CAPA for centOS distribution.

swan
2022-09-29 16:30:26

I can file a PR for this if no objection.
cc @kiran keshavamurthy @codenrhoden

kiran keshavamurthy
2022-09-29 19:54:36

@swan Feel free to open a PR

👍 swan
swan
2022-09-30 10:19:02

PR :
Is there any way I can inject this as env variable for generating AMIs, or do we need to wait for the release? If it’s latter, then we would be needing another release as soon as possible.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-30 10:21:54

You can provide the value as part of the —var-file

swan
2022-09-30 10:27:26

even ami specific filters? @Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-30 10:29:21

Oh sorry I think you're right. That var isn’t exposed. 😞 yeah, release needed

:sadblob: swan
Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-30 10:30:27

Thinking about it, might be worth updating that PR to use a user variable so it can be changed on the future

swan
2022-09-30 10:31:20

thats a good suggestion, we use these owner IDs as AMI filters in other OSes as well, I think there also we need the same update.

swan
2022-09-30 10:35:05

oh I see that amifilterowners is user variable

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-30 10:36:17

Oh nice! You should be able to override it then

swan
2022-09-30 10:36:31

yep will give it a try 🙂

swan
2022-09-30 10:39:00

worked 👍 thanks @Marcus Noble 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-30 10:40:08

Nice! Still worth getting a new release so it's fixed for everyone though.

swan
2022-09-30 10:40:42

Agreed!!! @codenrhoden @kiran keshavamurthy could a new release be taken care of?

swan
2022-09-30 12:03:40

we also need fix for cni versions used, image builder has hardcoded it to 0.8.x. A todo item before releasing new version.
Looks like its fixed in main branch

swan
2022-10-03 09:33:18

Bumping this thread again, could we get a newer release with newer changes as there are many fixes which could benefit the further image build for newer k8s releases

swan
2022-10-03 09:33:27

cc @codenrhoden @kiran keshavamurthy

kiran keshavamurthy
2022-10-03 17:56:40

I see 0.1.13 tag that @codenrhoden created recently.

swan
2022-10-04 06:30:39

I think that was created before this bug fix

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-04 07:13:50

The tag isn’t enough anyway. The container image also needs building and publishing.

:this: swan
swan
2022-10-06 08:57:17

@kiran keshavamurthy @codenrhoden @jsturtevant could you please help with the new release?

Kavitha Daula
2022-09-22 02:07:05

@Kavitha Daula has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-22 05:43:13

Looks like the 1.25.x builds are now failing due to the change of image registry

failed to pull image "k8s.gcr.io/coredns:v1.9.3"
Any chance we can get a new release? This fix is already in the main branch, just needs releasing 🙂

jsturtevant
2022-09-22 16:40:07

@kiran keshavamurthy are you able to help with releases? I haven't done one before

kiran keshavamurthy
2022-09-22 17:43:23

I’ve not as well.
Hello @codenrhoden, can you pls help us out here.

kiran keshavamurthy
2022-09-22 17:45:05

I believe Travis is the only person with the permission to cut a release. We should get a couple more folks added to that list. @jsturtevant if you are ok, I can open up a PR to get the 2 of added. Maybe @cecile too?

👍 jsturtevant
cecile
2022-09-22 17:47:40

I also haven’t done an image-builder release before, do we have docs on the process?

kiran keshavamurthy
2022-09-22 22:13:18

I don’t think so. Hopefully @codenrhoden is able to help out this time.

codenrhoden
2022-09-22 22:51:28

Hey folks. Definitely happy to help out however I can. I believe this is the location that would control being able to tag the repo: https://github.com/kubernetes/org/blob/44cf4faf10760dcc023dc4220b5e1a61875a93e1/config/kubernetes-sigs/sig-cluster-lifecycle/teams.yaml#L285

GitHub
codenrhoden
2022-09-22 22:52:57

And if you need me to tag, I can do that too. Just confirm the right commit - I haven’t been looking at the repo lately (as I’m sure you’re aware) 😆.

kiran keshavamurthy
2022-09-22 23:04:34

Hey Travis 👋
Thanks a lot.
I think we can tag it with the latest commit 4b97ae8b85216ac9e5f187fe88a2097e7813e525 unless someone has any objections.

codenrhoden
2022-09-23 15:56:00

I pushed a tag for v0.1.13. That should kick off the staging container build as well. Hope it all works out!

:thank_you: Marcus Noble, kiran keshavamurthy, jsturtevant
kiran keshavamurthy
2022-09-23 19:05:26

Thanks Travis, Can you pls document the steps somewhere so that we have it for future.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-26 06:49:12

@codenrhoden it looks like a tag has been created but not a release, the latest is still v0.1.12. Looks like because of that the docker image hasn't been created either. Is there still something that needs to be done to get the release out?

:this: swan
kiran keshavamurthy
2022-10-03 17:57:05

@codenrhoden ping ^^

codenrhoden
2022-10-04 20:42:21

@kiran keshavamurthy next steps would be to test out the staging builds and if all looks good, promote the image to prod. Sanika definitely knows how to do all that. You can make sure that the staging build worked correctly in test grid.

Sorry, I definitely thought people were aware of those next steps. Regardless, I will write it up. Should I just drop it in an issue, add it to the official docs, or put it in the repo wiki (which I don’t think we use)?

:thank_you: kiran keshavamurthy, jsturtevant
:ack: Sanika Gawhane
Sanika Gawhane
2022-10-04 22:18:40

Hey @codenrhoden 👋
I was able to build AMI and Azure image with the staging container. Testing OVA build atm.
Here's the PR for promoting 0.1.13 image -

cc - @kiran keshavamurthy

GitHub
🎉 codenrhoden
codenrhoden
2022-10-05 00:10:14

Cool! looks like it is already approved. dancing-penguin2

:thanks: Sanika Gawhane
:yes: Sanika Gawhane
🎉 Marcus Noble
timmycarr
2022-09-26 20:32:55

@timmycarr has joined the channel

quiquell
2022-09-27 09:34:19

@quiquell has joined the channel

quiquell
2022-09-27 09:34:30

Hi I have put in place a PR to pass console to kernel params to debian based images , how do I regenerate the image ?

GitHub
Shyam P R
2022-09-27 14:03:50

you have to execute the make command again

quiquell
2022-09-29 08:11:23

Working now, thanks

Cristiano Colangelo
2022-09-27 14:01:45

what are the default credentials to access a node through ssh?

Shyam P R
2022-09-27 14:03:35

Can you explain a bit more? which node? The node created during image building process?

Cristiano Colangelo
2022-09-27 14:04:10

images build under images/capi/packer

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-27 14:04:15

It varies depending on which provider you're building for (some don't set at all). For AWS see

GitHub
Cristiano Colangelo
2022-09-27 14:05:24

thanks

Shyam P R
2022-09-27 14:06:53

I dont think the ssh keys specified there are stored in the image so that when the an instance is created using the image, you can use the same ssh key. I may be wrong though. The ssh key is only valid for the instance which was used to create the image.

Cristiano Colangelo
2022-09-27 14:07:41

I tried to look in here (QEMU image) and I see some username and password but they don't work

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-27 14:08:11

Oh, do you mean the image when finished and not during the building?

Cristiano Colangelo
2022-09-27 14:08:33

yes

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-27 14:09:00

I think you'd want to handle that during the userdata stage, not during the building. I might be wrong though.

Cristiano Colangelo
2022-09-27 14:10:26

basically I'm trying to debug cloud-init errors... masters are provisioned on the infrastructure but fail at some point during initialization

Cristiano Colangelo
2022-09-27 14:10:40

maybe some DNS problem, not sure

Marcus Noble (k8s@marcusnoble.co.uk)
2022-09-27 14:11:47

oh, that's not idea. I'm not sure then unfortunately. 😞

Cristiano Colangelo
2022-09-27 14:12:31

there should be a way though 🤔

Shyam P R
2022-09-27 14:13:11

which provider?

Cristiano Colangelo
2022-09-27 14:13:20

OpenStack

Cristiano Colangelo
2022-09-27 14:14:32

will try thanks

Ray Krueger
2022-09-29 19:14:12

@Ray Krueger has joined the channel

knfoo
2022-10-04 08:04:15

@knfoo has joined the channel

Drew Hudson-Viles
2022-10-05 17:47:08

@Drew Hudson-Viles has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-10 10:02:02

Is anyone using image-builder with ubuntu 22.04? Any issues? I'm looking to update if it's nice and quick.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-10 15:54:54

So, looks like it's not possible to use image-builder in its current state with Ubuntu 22.04. Looks like a newer version of openssh comes by default that no longer allows for ssh-rsa keys to be used.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-10 15:58:25

Does anyone know what the process would be for updating the SSH key used by Ansible?
Doesn't look like I can override it so a new release would be required I think.

❤️ jayunit100-pub-chnl-plz
cecile
2022-10-11 22:42:34

@mboersma had a PR to add support for Azure

cecile
2022-10-11 22:43:01

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-11 08:34:21

☝️ Ok, I think I might have made progress on this. It looks like updating the version of Ansible (and goss, not sure why) has allowed me to get further. The new Ansible version isn't a major version bump so I assume it'll be ok to get a PR up for a new release.
Once I've confirmed the build finishes as expected I will look at getting a PR put up.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-11 10:53:25

PR to bump Ansible and goss versions:

GitHub
jayunit100-pub-chnl-plz
2022-10-11 12:26:40

Hi @Marcus Noble just wanted to say thanks for that ... noticed the old ansible version recently also but I was too lazy to do anything about it. And yes ubuntu 22 thank you for that also. Cc @kiran keshavamurthy

👍 kiran keshavamurthy
Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-11 12:28:27

I'm just testing out ubuntu 22.04 builds for CAPA and CAPG. If all goes well I'll have another PR up with build tasks for those.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-11 16:33:46

PR adding Ubuntu 22.04 support for AWS and GCP - (requires the above PR to be merged)

GitHub
mimmus
2022-10-12 14:09:50

@mimmus has left the channel

knfoo
2022-10-12 14:19:03

It is great to see all the work on adding Ubuntu 22.04 support 🎉
We are using these images with metal3 any work ongoing for the Ubuntu 22.04 for the make target build-qemu-ubuntu-2004-efi so it becomes build-qemu-ubuntu-2204-efi ?
I tried looking for it in the issues list but was not able to

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-13 13:41:12

Hey @knfoo I've only been focussing on AWS and GCP for my changes as it's what we need currently but we'll also be needing the qemu targets soon for use with OpenStack.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-13 13:41:31

I'm hoping the changes are basically the same but haven't tried it out yet.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-13 14:40:30

If you happen to have the time and means to test out this PR I think it might be all that's needed:
I'm currently unable to test locally on my ARM-based Mac 😞

GitHub
knfoo
2022-10-14 07:42:17

Hey @Marcus Noble
This is great - i will try it out after my vacation.

:thank_you: Marcus Noble
Joe Kratzat
2022-10-18 02:04:08

I was attempting to build a windows image today and I’m getting the following error:

fatal: [default]: FAILED! => {"attempts": 5, "changed": false, "dest": "c:\k\nssm.exe", "elapsed": 0.21601959999999998, "msg": "Error downloading '' to 'c:\k\nssm.exe': The remote server returned an error: (404) Not Found.", "status_code": 404, "url": ""}
When I attempt to go to I see the 404. It seems the URL is set here

What am I doing wrong?

Joe Kratzat
2022-10-18 17:43:11

hmmm seems like that url is bad. I downloaded nssm.exe and placed it in another public place and the image build worked

jsturtevant
2022-10-18 19:19:23

we are aware of this, you have found the mitigation. We are working on addressing it

Joe Kratzat
2022-10-18 19:20:21

awesome thank you. I’ll use the mitigation simple_smile

jsturtevant
2022-10-18 19:46:17

I know you saw it on the other thread, dropping this here for posterity:

GitHub
:thank_you: Joe Kratzat
Ivan Šumak
2022-10-19 19:28:01

@Ivan Šumak has joined the channel

Portia
2022-10-20 00:05:10

@Portia has joined the channel

Joe Kratzat
2022-10-20 13:46:45

Hey folks.

I’m trying to understand how passwords are set for winRM for CAPI provider images. I see the OVA and VBOX are hardcoding winrm_password in their windows packer config, but AWS and Azure aren’t. How are these later images setting the passwords at image build time?

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-24 07:02:26

Is anyone free to take a look at for me please? 🙏

GitHub
Vignesh Goutham
2022-10-24 18:49:32

Hello all, whats the current topic request format for office hours? If there is a doc, can someone point me to it? This one seems to be old. Or do we just post here the day before?

Joe Kratzat
2022-10-24 20:43:00

Hey all, I’m not sure what I’ve done to have the pull-ova-all tests failing in my pr

but I’m seeing the following

 'packer' has been installed to /home/prow/go/src/sigs.k8s.io/image-builder/images/capi/.local/bin, make sure this directory is in your $PATH
hack/ensure-goss.sh
/root/.packer.d/plugins/packer-provisioner-goss: OK
hack/ensure-ovftool.sh
rockylinux-8: FAILED. See logs in the artifacts folder.
ubuntu-2004: FAILED. See logs in the artifacts folder.
photon-3: FAILED. See logs in the artifacts folder.
flatcar: FAILED. See logs in the artifacts folder.
ubuntu-1804: FAILED. See logs in the artifacts folder.
centos-7: FAILED. See logs in the artifacts folder.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:-:-- --:-:-- --:-:-- 0
0 0 0 0 0 0 0 0 --:-
:-- --:-:-- --:-:-- 0

1 7400k 1 111k 0 0 204k 0 0:00:36 --:-:-- 0:00:36 204k
100 7400k 100 7400k 0 0 7815k 0 --:-
:-- --:-:-- --:-:-- 17.7M
govc: Post "": dial tcp 10.2.224.4:443: i/o timeout
TIA for any help simple_smile

Chris Privitere
2022-10-25 15:55:25

@Chris Privitere has left the channel

Jamie Monserrate
2022-10-27 02:39:37

@Jamie Monserrate has left the channel

Remi Le Trocquer
2022-10-27 09:44:01

@Remi Le Trocquer has joined the channel

Sriraman Srinivasan
2022-10-28 06:45:18

@kiran keshavamurthy Reg: PR 1003, the new ubuntu versions use cloud init to boot and it expects the drive to be labeled as cidata to work. But for some reason setting floppylabel to the value does not seem to take effect. As a result I see only loading the kickstart files (meta-data and user-data) as files on cdrom(able to set the cdlabel tag and gets reflected). However for this to work either xorriso or mkisofs needs to be available to create the cdrom which then can be mounted. Adding either of those will need to change prow job container running the build. So need couple of inputs here

  • Is it ok to go ahead with using cdrom as mount device for kiskstart files or do you see any other way to do it

  • Also can you please point me to the prow job where the image will be build

GitHub
Sriraman Srinivasan
2022-11-04 04:14:42

@kiran keshavamurthy can you please suggest here.

kiran keshavamurthy
2022-11-14 17:42:01

Thanks for the patience @Sriraman Srinivasan. Been a bit busy and was on PTO. Will look into this soon.

👍 Sriraman Srinivasan
k8scapv
2022-10-28 12:42:17

@k8scapv has joined the channel

k8scapv
2022-10-28 13:13:50

any readily available rhel 8 ova vm template for capv ?

swan
2022-10-28 16:13:17

Hey Folks, wanted to know if there is something changed recently from v0.1.13 regarding AMI generation? As the names that are now generated for CAPA AMIs by image builder are as capa-ami-amazon-2-v1.25.3-1665727783 (notice suffix v in k8s release) as opposed to what it was in previous releases capa-ami-amazon-2-1.25.2-00-1664536077 . This is causing issues in CAPA for pushing AMI images, as there is some strict checking of formatting. We could resolve this on CAPA side, but just wanted to make sure if this change was intentional or was done by mistake?

swan
2022-10-28 16:13:50

cc @Skarlso @richcase @Daniel Lipovetsky

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-28 16:15:32

I'm pretty sure it was intentional to be inline with other providers. I recently looked up the change. Give me a min and I'll dig it out

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-28 16:16:55



Released in

GitHub
GitHub
swan
2022-10-28 16:17:48

thanks @Marcus Noble this helps, I would make change in CAPA accordingly.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-28 16:18:53

Yeah, I already approved your PR in CAPA 🙂

swan
2022-10-28 16:19:06

needs more change 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2022-10-28 16:19:27

Ping me when done 🙂 Happy to review again

swan
2022-10-28 16:19:41

thanks a lot, surely will do 🙂

Peter Bücker
2022-11-08 21:11:36

@Peter Bücker has joined the channel

k8scapv
2022-11-09 05:55:05

can we use existing OVA image (already existing one which is verified by security team) instead of iso and install addons (k8s,cni,container stuff to make it capi v conformant image) on top of that OVA using image builder and create a vm template out of it.Can some one guide me on this

subhasmita
2022-11-09 06:34:19

@subhasmita has left the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-10 13:51:51

Do the image-builder office hours still happen? Doesn't look like there's been one since July according to the meeting notes.

Joe Kratzat
2022-11-10 13:55:56

I believe if people don’t put things on the agenda they will cancel the meeting

If no Agenda items are present the night before, the meeting will be canceled.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-10 13:57:57

I haven't seen any messages about it being canceled either though. Which makes me wonder if it's just kinda died off.

Joe Kratzat
2022-11-10 13:59:42

yeah I’m not sure there. Last thing about office hours was from

Cecile Robert-Michon (https://kubernetes.slack.com/team/U98JPHB2M)
Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-10 14:07:39

I might try adding topics for next week and see what happens I guess 😆

😅 Joe Kratzat
cecile
2022-11-10 23:19:10

@codenrhoden used to host these but he had to step away from the project

cecile
2022-11-10 23:20:04

@mboersma @jsturtevant @kiran keshavamurthy and I have been maintaining the project on a best effort basis (basically just keeping the lights on)

cecile
2022-11-10 23:20:27

I personally have a conflict with the current 8am time and can’t make it most weeks

cecile
2022-11-10 23:21:30

the project is a bit low on maintainers right now so if anyone has interest in stepping up to host the office hours please do

Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-11 07:14:07

That's actually one of the things I wanted to discuss 🙂 image-builder is a pretty important project for us at Giant Swarm and would love if we could help out . There's a lot of improvements / fixes we'd like to see but the current level of activity on the project makes that difficult. I'll get some topics added to the agenda for next week later today 🙂

❤️ jsturtevant
richcase
2022-11-14 07:29:26

We are in a similar position as well, image builder is pretty important to us as well. We also want to make improvements/changes.

🎉 Marcus Noble
kiran keshavamurthy
2022-11-14 17:39:35

Image-builder is important to us as well. I have to split time between upstream and downstream work so sometimes upstream work takes a hit.
I’m completely on board with changing the office hours to better suit everyone and love to get more maintainers

👍 cecile, jsturtevant
Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-17 16:01:17

Just wanted to double check others are joining the office hours. I'm currently waiting in the call. 🙂

:this: MarcelMue
Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-17 16:07:37

@kiran keshavamurthy @richcase @cecile Are any of you planning to join?

richcase
2022-11-17 16:08:07

I am, sorry

cecile
2022-11-18 00:07:20

I personally have not had the bandwidth to be active enough as image-builder maintainer lately and I think it's time I officially step down to make space for new folks

cecile
2022-11-18 00:13:21

I will make a PR soon to officially move myself to emeritus. I also nominate @mboersma as new maintainer, he's been a lot more involved in the project than I have lately and is interested in helping. cc @kiran keshavamurthy

:ack: kiran keshavamurthy
jsturtevant
2022-11-21 16:58:55

I can no longer make the 8am meetings on Thursdays but would be open to meeting another time. Should we get a doodle poll going for new time as it seems we have a bunch on new folks?

Marcus Noble (k8s@marcusnoble.co.uk)
2022-12-01 22:54:05

Thanks @cecile 🙂 I've opened a PR to add myself to the reviewers:

GitHub
🎉 cecile
aniruddha
2022-11-12 05:13:18

@aniruddha has left the channel

Kemalettin
2022-11-12 15:14:33

@Kemalettin has joined the channel

Magnus RC
2022-11-14 12:39:24

@Magnus RC has joined the channel

willie
2022-11-14 21:33:28

@willie has joined the channel

swan
2022-11-16 11:46:00

Hey folks, since the newer k8s release versions are compatible with containerd version 1.6.4+, image builder still uses default as 1.6.2.
I tried below giving parameters such that we use compatible version of containerd while generating CAPA AMIs:

"containerd_version": "1.6.5",
"containerd_checksum": "cf02a2da998bfcf61727c65ede6f53e89052a68190563a1799a7298b0cea86b4",
"containerd_url": ""
but its failing with below error:
fatal: [default]: FAILED! => {"changed": true, "checksum_dest": null, "checksum_src": "8b354c7fcc59c66ce8ade0bc137782838709fa3c", "dest": "/tmp/containerd.tar.gz", "elapsed": 0, "msg": "The checksum for /tmp/containerd.tar.gz did not match 91f1087d556ecfb1f148743c8ee78213cd19e07c22787dae07fe6b9314bec121; it was cf02a2da998bfcf61727c65ede6f53e89052a68190563a1799a7298b0cea86b4.", "src": "/tmp/.ansible/ansible-tmp-1668598896.3124301-85962-228207629295306/tmpps0mndg2", "url": ""}
Although I see checksum set is correct but i think somehow this is interfering with other attribute related to CRI.

swan
2022-11-16 11:47:32

oh i think the attribute should be containerd_sha256 facepalm

swan
2022-11-16 11:49:18

although at some point, we must update this version to higher containerd version compatible with recent k8s versions

swan
2022-11-16 12:11:18

also even after setting right paramter I am getting below error

  ubuntu-20.04: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to create temporary directory. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p \"echo /tmp/.ansible\"&& mkdir \"echo /tmp/.ansible/ansible-tmp-1668600507.193056-96724-263289874289379\" && echo ansible-tmp-1668600507.193056-96724-263289874289379=\"echo /tmp/.ansible/ansible-tmp-1668600507.193056-96724-263289874289379\" ), exited with result 1", "unreachable": true}
could someone help solve this

swan
2022-11-17 05:28:46

cc @Marcus Noble if you know about this

Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-17 07:04:15

Sorry, I was AFK yesterday. 🙂 Do you get the same error without making the changes to containerd? I doesn't look like it should be related from what I can tell. I can try and dig in a bit more later today and see if I can see the cause.

richcase
2022-11-17 07:24:59

I can try this morning

swan
2022-11-17 07:25:47

I wanted to change containerd version thats why trying this, but didnt get this error while not specifying containerd versions etc.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-17 07:26:24

Was the 3 containerd values the only thing you changed?

swan
2022-11-17 07:27:32

{
"kubernetes_series": "1.25",
"kubernetes_semver": "1.25.4",
"kubernetes_rpm_version": "1.25.4-0",
"kubernetes_deb_version": "1.25.4-00",
"kubernetes_source_type": "pkg",
"kubernetes_http_source": "",
"kubernetes_rpm_repo": "",
"kubernetes_rpm_gpg_key": "\" \"",
"kubernetes_rpm_gpg_check": "True",
"kubernetes_deb_repo": "\" kubernetes-xenial\"",
"kubernetes_deb_gpg_key": "",
"kubernetes_container_registry": "registry.k8s.io",
"kubernetes_load_additional_imgs": "false",
"kubeadm_template": "etc/kubeadm.yml",
"containerd_version": "1.6.6",
"containerd_sha256": "0212869675742081d70600a1afc6cea4388435cc52bf5dc21f4efdcb9a92d2ef",
"containerd_url": ""
}
This is the env vars i generally use to generate AMIs, only thing added here is containerd stuff

swan
2022-11-17 08:30:06

ok made some progress, looks like i was using wrong containerd url, so removed url and it started working but failed at this point

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.errors.AnsibleFilterError: Version comparison failed: '<' not supported between instances of 'int' and 'str'
flatcar-stable: fatal: [default]: FAILED! => {"changed": false, "msg": "AnsibleFilterError: Version comparison failed: '<' not supported between instances of 'int' and 'str'"}
I am not sure if this version is supported or not, ideally i should be able to build with any containerd version

Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-17 08:39:47

Any idea what task is throwing that error? Struggling to find where the version is checked 😕

swan
2022-11-17 08:40:07

sorry missed it 😄

 TASK [containerd : Copy in containerd config file etc/containerd/config.toml] **

Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-17 08:42:08

😕 That doesn't seem to do anything with the version.

Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-17 08:42:51

Mind posting the ~5 tasks that completed in the run up to that error?

Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-17 08:44:32

I think it might be complaining about a different "version" in that error. As far as I can see, the containerd_version is used for 2 things - building the URL (which you're overwriting anyway) and by goos to check the right version was installed.

swan
2022-11-17 09:01:18

here you go

flatcar-stable: TASK [include_role : containerd] *
flatcar-stable:
flatcar-stable: TASK [containerd : download containerd] *

flatcar-stable: changed: [default]
flatcar-stable:
flatcar-stable: TASK [containerd : Create a directory if it does not exist] *
flatcar-stable: ok: [default]
flatcar-stable:
flatcar-stable: TASK [containerd : unpack containerd for Flatcar to /opt/bin] *

flatcar-stable: changed: [default]
flatcar-stable:
flatcar-stable: TASK [containerd : delete /opt/cni directory]

flatcar-stable: changed: [default]
flatcar-stable:
flatcar-stable: TASK [containerd : delete /etc/cni directory] *
flatcar-stable: changed: [default]
flatcar-stable:
flatcar-stable: TASK [containerd : Creates unit file directory] *

flatcar-stable: changed: [default]
flatcar-stable:
flatcar-stable: TASK [containerd : Create systemd unit drop-in file for containerd to run from /opt/bin] *
flatcar-stable: changed: [default]
flatcar-stable:
flatcar-stable: TASK [containerd : Create containerd memory pressure drop in file]

flatcar-stable: changed: [default]
flatcar-stable:
flatcar-stable: TASK [containerd : Create containerd max tasks drop in file] *

flatcar-stable: changed: [default]
flatcar-stable:
flatcar-stable: TASK [containerd : Create containerd http proxy conf file if needed] *
flatcar-stable: changed: [default]
flatcar-stable:
flatcar-stable: TASK [containerd : Creates containerd config directory] *

flatcar-stable: changed: [default]
flatcar-stable:
flatcar-stable: TASK [containerd : Copy in containerd config file etc/containerd/config.toml]

flatcar-stable: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ansible.errors.AnsibleFilterError: Version comparison failed: '<' not supported between instances of 'int' and 'str'
flatcar-stable: fatal: [default]: FAILED! => {"changed": false, "msg": "AnsibleFilterError: Version comparison failed: '<' not supported between instances of 'int' and 'str'"}
flatcar-stable:
flatcar-stable: PLAY RECAP
**
flatcar-stable: default : ok=34 changed=25 unreachable=0 failed=1 skipped=145 rescued=0 ignored=0
flatcar-stable:

Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-17 09:40:49

ah ha! I think it's this line: https://github.com/kubernetes-sigs/image-builder/blob/02df45969409c7f18f2cf7e63b70[…]i/ansible/roles/containerd/templates/etc/containerd/config.toml
Can you try setting kubernetes_semver to be v1.25.4 (hopefully that doesn't break elsewhere)

GitHub
swan
2022-11-17 09:41:25

Oh great i will give it a try

swan
2022-11-17 09:54:58

oh atleast the previous failure is gone now 😅 thanks @Marcus Noble I will ping back if I get more errors 😉

Marcus Noble (k8s@marcusnoble.co.uk)
2022-11-17 10:01:57

🎉

k8scapv
2022-11-18 06:50:21

can i run ansible scripts alone on the OVA from the image builder repo ? can some one guide on this.

k8scapv
2022-11-18 06:51:12

I do not want image builder to create an ova for me ,i have an ova and i just want to install rest all things on top of that ova,so that i can use that ova to create vmtemplate

k8scapv
2022-11-18 06:51:25

can some one guide how can i do that

jsturtevant
2022-11-21 17:08:09

there isn't a guide but if you have the vm booted you can configure Ansible to connect to that VM then run the ansible scripts in . You will have to set all the ansible variables yourself

GitHub
Julien Klaer
2022-11-22 17:05:37

@Julien Klaer has joined the channel

Joe Kratzat
2022-11-29 13:01:19

Hey all,

I have a PR failing tests around pull-azure-sigs

I see

 sig-flatcar: FAILED. See logs in the artifacts folder.
sig-flatcar-gen2: FAILED. See logs in the artifacts folder.
in the logs, but not seeing why flatcar is failing.

Jeremi Piotrowski
2022-11-29 15:53:43

im on it

:thank_you: Joe Kratzat, mboersma
Joe Kratzat
2022-11-29 19:55:52

any luck?

Jeremi Piotrowski
2022-11-30 11:16:26

it's merged

GitHub
Ricky Sadowski
2022-11-30 18:19:37

@Ricky Sadowski has joined the channel

Vibhor Chinda
2022-11-30 18:38:44

@Vibhor Chinda has joined the channel

cpanato
2022-12-02 10:23:11

seeking a lgtm here 🙂

GitHub
👍 jsturtevant
Vibhor Chinda
2022-12-05 16:09:45

Hii everyone :)) 👋
I attended the last capi office hours and found out that this project needs some contributors who can help.

I am a beginner in this space but am looking to contribute wherever and whatever I can.

I would love to know more about the project and can help with beginner friendly issues if there are any 👀.

Any kind of guidance will be great :))
thanks

jsturtevant
2022-12-05 17:15:23

welcome! We have a book to get started: . Triaging issues could be a good way to get started getting familiar with the type of work in the project. Feel free to ask questions

:ty: Vibhor Chinda
Vibhor Chinda
2022-12-06 06:37:14

Thanks @jsturtevant
I will have a look at the book :))

Vivek Koppuru
2022-12-07 08:33:19

@Vivek Koppuru has joined the channel

Esme
2022-12-16 01:16:16

@Esme has joined the channel

swan
2022-12-19 13:45:35

Hey folks, I am trying to build CAPA AMI images, but I am getting following ansible ssh error, could someone help around fixing this issue? This has been specifically started occurring recently as it used to work before. Maybe it has something to do with mac OS update, but I am not sure

amazon-2: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Unable to negotiate with 127.0.0.1 port 55506: no matching host key type found. Their offer: ssh-rsa", "unreachable": true}

nikparasyr
2022-12-19 13:46:25

there are a couple of issues and PRs open for this

swan
2022-12-19 13:47:04

oh thats good to know, will wait for the fix. Thankyou @nikparasyr

nikparasyr
2022-12-19 13:47:23

maybe this is the one blocking you?

GitHub
swan
2022-12-19 13:47:44

yes that’s right

swan
2022-12-19 13:48:18

I ll give it a try with args

nikparasyr
2022-12-19 13:48:26

there is a comment there on how to bypass it. The PRs to fix it have been open for a while AFAIK, so i just had to use what is mentioned in the comment to move forward. Hope it works for you

👍 swan
Sai Preetham Bojja
2022-12-19 15:56:20

@Sai Preetham Bojja has joined the channel

Alex Hernandez
2022-12-19 15:57:47

@Alex Hernandez has joined the channel

quba
2022-12-28 16:10:28

@quba has joined the channel

bkc
2022-12-30 15:59:51

@bkc has joined the channel

Aravind Ravichandran
2023-01-04 10:27:09

@Aravind Ravichandran has joined the channel

Aravind Ravichandran
2023-01-04 10:27:58

Hi team , need help on creating clusterapi image for kubevirt

Julien Klaer
2023-01-05 08:37:05

Hello there 👋 Happy new year!
Is there any chance this issue makes sense? Would it make sense to define the end goal of it? I can try to work something out for this

jsturtevant
2023-01-09 17:03:01

this would avoid passing it to the image builder, but would still need it in scripts? I think it probably makes sense to minimize passing hte variables around. @mboersma any thoughts?

jdetiber
2023-01-12 15:42:27

@jdetiber has left the channel

Paokrab
2023-01-16 17:35:05

@Paokrab has joined the channel

Amim Knabben
2023-01-17 14:58:20

Hey, did already tried to run image-builder on a pre-existent image? More specifically decouple and reuse the ansible roles on a running VM.

vrabbi
2023-01-17 17:21:35

Ive done it

:partyk8s: Amim Knabben
Alessandro Giorgio Togna
2023-01-19 21:22:26

@Alessandro Giorgio Togna has joined the channel

Thorben
2023-01-21 13:06:53

@Thorben has joined the channel

Thorben
2023-01-21 13:14:07

Hey! With the latest minor releases of the 24, 25 and 26 series I'm running into build errors for GCP cluster-api images. 1.24.9, 1.25.5 and 1.26.0 work just fine, with 1.24.10, 1.25.6 and 1.26.1 I get the same error across all 3 builds:

ubuntu-2204: TASK [kubernetes : Install Kubernetes] **
ubuntu-2204: fatal: [default]: FAILED! => {"cache_update_time": 1674304180, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\" install 'kubelet=1.24.10-00' 'kubeadm=1.24.10-00' 'kubectl=1.24.10-00' 'kubernetes-cni=1.1.1-00'' failed: E: Unable to correct problems, you have held broken packages.\n", "rc": 100, "stderr": "E: Unable to correct problems, you have held broken packages.\n", "stderr_lines": ["E: Unable to correct problems, you have held broken packages."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nSome packages could not be installed. This may mean that you have\nrequested an impossible situation or if you are using the unstable\ndistribution that some required packages have not yet been created\nor been moved out of Incoming.\nThe following information may help to resolve the situation:\n\nThe following packages have unmet dependencies:\n kubeadm : Depends: kubernetes-cni (>= 1.2.0)\n kubelet : Depends: kubernetes-cni (>= 1.2.0)\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Some packages could not be installed. This may mean that you have", "requested an impossible situation or if you are using the unstable", "distribution that some required packages have not yet been created", "or been moved out of Incoming.", "The following information may help to resolve the situation:", "", "The following packages have unmet dependencies:", " kubeadm : Depends: kubernetes-cni (>= 1.2.0)", " kubelet : Depends: kubernetes-cni (>= 1.2.0)"]}

I'm assuming other providers are also affected, but couldn't find anything in the issues. That is, if this file is actually responsible for installing the kubernetes-cni .deb package:

{
...
"kubernetes_cni_deb_version": "1.1.1-00",
---
}

Is there something I'm doing wrong or did this simply not pop up in the nightly CI yet?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-01-23 07:28:21

I was able to build it with these values:

{
"kubernetes_cni_deb_version": "**",
"build_name": "ubuntu-2204",
"distribution_release": "jammy",
"distribution_version": "2204"
}
This ensures the latest deb version is used from the package manager.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-01-23 07:32:55

The latest nightly run is here: Prow Job
This runs this script: ci-gce-nightly.sh
I don't know much about it but it seems to use this for the 1.26 builds which is still targeting the 1.26.0 version - overwrite-1-26.json

Thorben
2023-02-12 10:00:05

Issue has been resolved in this commit, by the looks of it - I can build the versions mentioned initially now:

puneetk
2023-01-26 06:28:22

@puneetk has joined the channel

Sakari Poussa
2023-01-26 13:57:26

@Sakari Poussa has joined the channel

Joe Kratzat
2023-02-08 13:55:14

We have run into a few times where users are getting errors because the IBVERSION ENV isn’t set when trying to build an image for our packer config. I see other configs using the same “ib_version”: “{{envIBVERSION}}“, but nothing in the book calls out setting this. How are others 1) setting this environment variable for the user or 2) informing the users to set it?

Jon Zeolla
2023-02-14 18:05:55

@Jon Zeolla has joined the channel

tomoyuki
2023-02-15 08:00:17

@tomoyuki has joined the channel

fc
2023-02-23 08:11:24

@fc has joined the channel

fc
2023-02-23 08:45:16

Hi Everyone,

I just started experimenti with image-builder to build our own CAPZ Flatcar images.

for capz the init-sig.sh script is run to create the image definition but the format is very hardcoded and not ideal ( at least for our use case )

the generated format is capi-flatcar-stable-$(FLATCARVERSION)-gen2 while ideally i would like, similar to what the CAPI images are capi-flatcar-stable-$(KUBERNETESVERSION) so i can then have an image inside of it for each version of flatcar

➜ az sig image-definition list-community --public-gallery-name flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0 --location westeurope | jq '.[].name'
"flatcar-stable-amd64-capi-v1.23.13"
"flatcar-stable-amd64-capi-v1.24.6"
"flatcar-stable-amd64-capi-v1.24.9"
"flatcar-stable-amd64-capi-v1.25.4"
"flatcar-stable-amd64-capi-v1.25.6"
"flatcar-stable-amd64-capi-v1.26.0"

➜ az sig image-version list-community --location westeurope --public-gallery-name flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0 --only-show-errors --gallery-image-definition flatcar-stable-amd64-capi-v1.24.9 -o table
ExcludeFromLatest Location Name PublishedDate UniqueId
------------------- ---------- -------- -------------------------------- --------------------------------------------------------------------------------------------------------------------------------
True westeurope 3374.2.1 2023-01-06T00:29:51.344093+00:00 /CommunityGalleries/flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0/Images/flatcar-stable-amd64-capi-v1.24.9/Versions/3374.2.1

  • how does the CAPI project achieve this using the same init-sig.sh script that is in the repo ?


thanks !

fc
2023-02-23 09:25:47

@cecile in case you know about it , i did not want to cross post in cluster-api-azure

🙏

fc
2023-02-23 11:43:33

for reference

Jeremi Piotrowski
2023-02-24 14:58:55

hi @fc. what we do for Flatcar is we use image-builder to build the image, and then republish into a seperate gallery that has the desired structure that we want. The republishing is done by using the gallery-image-version-id as the image source

fc
2023-02-27 08:10:54

I see, thanks !

fc
2023-02-27 15:45:28

@Jeremi Piotrowski i have a follow up question to this 🙂

I am building directly with image-builder into a Community gallery and is working fine now but , in order to use my image, i need to specify something like

      image:
computeGallery:
gallery: test-xxxx-820f-b52ca78f96e6
name: capi-flatcar-stable-1.24.9-gen2
plan:
offer: flatcar-container-linux-free
publisher: kinvolk
sku: stable-gen2
version: latest

but when i use the flatcar4capi gallery i don't have to specify the plan section.

is this difference because of your process of republishing into a separate gallery ?

thanks

Jeremi Piotrowski
2023-02-27 15:52:37

@Mateusz Gozdek (invidian) there is some magic here right?

fc
2023-02-27 15:53:07

i like magic 🎉

Jeremi Piotrowski
2023-02-27 15:53:16

or do we build the community gallery image with a community gallery image as a source?

fc
2023-02-27 15:54:09

my concern is that

  • i did a lot of testing using the flatcar4capi images

  • i am moving onto building my images from image-builder

  • to use flatcar4capi images i don't need to accept terms of the parent image like i do with my build ( and i have to accept the parent flatcar terms ... which is ok , but is different and i want to understand why )

  • maybe they are actually not equivalent somehow ?

Mateusz Gozdek (invidian)
2023-02-27 16:22:05

Hello. Images in flatcar4capi are build from Flatcar VHDs imported into a SIG, so their advantage is that they don't require plan information. That's the big part of it.

There is also flatcar community gallery, which you can use as a source for building your images using image-builder. Let me dig up some sample JSON packer values we use for building the images.

fc
2023-02-27 16:22:47

yeah that make sense. thanks

There is also flatcar community gallery, which you can use as a source for building your images using image-builder. Let me dig up some sample JSON packer values we use for building the images.
that is what i am doing 👍 through image-builder

Mateusz Gozdek (invidian)
2023-02-27 16:23:08

From our release automation:

  cat <{
"sig_image_version": "${FLATCAR_VERSION}",
"kubernetes_semver": "${KUBERNETES_SEMVER}",
"image_name": "${IMAGE_NAME}",
"image_offer": "",
"image_publisher": "",
"image_sku": "",
"image_version": "",
"plan_image_offer": "",
"plan_image_publisher": "",
"plan_image_sku": "",
"source_sig_subscription_id": "${AZURE_SUBSCRIPTION_ID}",
"source_sig_resource_group_name": "${STAGING_SIG_RESOURCE_GROUP}",
"source_sig_name": "${FLATCAR_STAGING_GALLERY_NAME}",
"source_sig_image_name": "${FLATCAR_IMAGE_NAME}",
"source_sig_image_version": "${FLATCAR_VERSION}"
}
EOF

fc
2023-02-27 16:24:18

now i am just trying to validate the differences between the nodes i get with flatcar4capi and from image-builder
built images.


I can see that with the image from flatcar4capi i get containerd 1.6.14 while on the one from image-builder i get 1.6.2 ... do you customize / override that ?

Mateusz Gozdek (invidian)
2023-02-27 16:25:35

No, we build from master branch of image-builder. I don't remember, but maybe in Flatcar it's using baked in containerd version as opposed to one installed by Ansible? So maybe the difference is Flatcar version used?

fc
2023-02-27 16:27:11

thanks for confirming that.

i am going through the ansible code and our pipelines now , maybe we are just out of date or something.

i can see the playbook eventually run does indeed specify 1.6.2

 sig-flatcar-gen2: Executing Ansible: ansible-playbook -e packer_build_name="sig-flatcar-gen2" -e packer_builder_type=azure-arm --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url= containerd_sha256=91f1087d556ecfb1f148743c8ee78213cd19e07c22787dae07fe6b9314bec121 pause_image=k8s.gcr.io/pause:3.6 containerd_additional_settings= containerd_cri_socket=/var/run/containerd/containerd.sock containerd_version=1.6.2
now i just need to walk back and find where and why those are set to 1.6.2 🙂

thanks foryour help 🙏

👍 Mateusz Gozdek (invidian)
fc
2023-02-27 16:34:58

simple, i am building from tag 01.13 ( which indeed uses containerd 1.6.2 still ) and master is using the upgraded version



🙂 thanks again

👌 Mateusz Gozdek (invidian)
Mateusz Gozdek (invidian)
2023-02-27 16:37:51

Cool. Feel free to ping me if you have some further questions. I was mainly driving the work on community SIG for Flatcar and I'm always happy to help (or redirect to team members which might be more knowledgeable than me) 🙂

fc
2023-02-27 16:38:18

thanks will do 🙏

fc
2023-02-28 11:37:54

hi @Mateusz Gozdek (invidian) sorry to ping you again, one quesiton

you said you build VHD using image-builder then import into SIG

i wanted to try the same but when i look at i don't see flatcar in the lsit of VHD supported targets.

do you have patch or something on top of image builder ?

if your code for the build pipeliens is available on github i'ld love to take a look 🙂
thanks 🙏

Mateusz Gozdek (invidian)
2023-02-28 11:45:43

The script is not public yet, as it's a first version and we didn't put much effort into it yet, but yeah, eventually it will be public. This is how it looks: .

As you won't have access to storage account which holds Flatcar VHD images, You probably need to download a VHD from the Flatcar release, upload it to storage account, and then you should be able to use the script above.

It's nothing fancy really.

fc
2023-02-28 11:46:04

thanks 🙏

fc
2023-02-28 17:42:50

unrelated , to an extent , question.

do you know why the .4 of the stable channel does not exist in azure as a vm image yet ?

➜ az vm image list --publisher kinvolk --sku stable-gen2 --offer flatcar-container-linux-free --all -o json | jq -r '[.[].version] | sort_by( values | split(".") | map(tonumber) ) | .[-1]'
3374.2.3

Mateusz Gozdek (invidian)
2023-02-28 17:46:36

I guess the image has not been approved yet, which is odd. Maybe @Kai Lüke or @Jeremi Piotrowski know what's the status of it?

BTW, is the image available in community gallery? I can make it so if it's also missing.

fc
2023-02-28 17:47:36

have not checked but can do that

fc
2023-02-28 17:49:04

in the flatcar4capi i can only see .1 , let me see to find the name of the flatcar community gallery

fc
2023-02-28 17:52:41

in the flatcar one i see the same, .3 as latest

 λ az sig image-version list-community --location westeurope --public-gallery-name flatcar-23485951-527a-48d6-9d11-6931ff0afc2e --only-show-errors --gallery-image-definition flatcar-stable-amd64 | jq '.[].name'
"3374.2.0"
"3374.2.1"
"3374.2.3"

👍 Mateusz Gozdek (invidian)
Mateusz Gozdek (invidian)
2023-02-28 17:53:55

I'll try to push the latest image there.

fc
2023-02-28 17:54:34

nice thanks 🙏 i will need to see to change my pipeline to use the gallery rather than the vm image as source

👍 Mateusz Gozdek (invidian)
Mateusz Gozdek (invidian)
2023-02-28 18:34:44

.4 image should be there already.

fc
2023-02-28 18:35:06

Thanks 🙏

Jeremi Piotrowski
2023-02-28 18:41:43

Yeah azure publishing got delayed this time round and is taking longer as well

fc
2023-03-01 10:38:17

@Mateusz Gozdek (invidian) thanks for all your help, i switched to the community gallery "/CommunityGalleries/flatcar-23485951-527a-48d6-9d11-6931ff0afc2e/Images/flatcar-stable-amd64/Versions/3374.2.4"

just a question, how official is this gallery when compared to the flatcar Marketplace offer ?

Mateusz Gozdek (invidian)
2023-03-01 10:42:49

It's official (), but you would probably be an early adopter. We hope it will enable Flatcar users on Azure to get faster access to latest image versions (because of easier and more automated release process) and without requirement of accepting plans.

Right now image publishing there is not wired to the CI, so there might be still a delay until the images show up, but when we see people using it, I'm sure CI will be prioritized to have the process fully automated.

GitHub
fc
2023-03-01 10:45:44

nice thanks

without requirement of accepting plans.
so using the Community Gallery as my source rather than the Martketplace offer means i don't have to accept terms in every subscription ? nice

We hope it will enable Flatcar users on Azure to get faster access to latest image versions
👍 exactly why i switched

I am ok to be an early adopter as we are not in production with capz yet and i'd rather switch the sooner the better and give feedback if needed

Right now image publishing there is not wired to the CI
'm sure CI will be prioritized to have the process fully automated.
🙏 🤞

I might come here and ask for new images when i see releases in the flatcar RSS feed 🙂

thanks

Mateusz Gozdek (invidian)
2023-03-01 10:47:38

I'll make sure images are published as soon as I see a feed as well then. It's probably time to enable replication to all regions then as well.

👍 fc
fc
2023-03-01 10:48:03

Right now image publishing there is not wired to the CI
I do wonder though if ,when that time comes, i should stop building my own images ... i mean here https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/2890/files?short_path=91b8a4f#diff-91b8a4f39cd0f7ee28f[…]67069103bd0fa528e84cb4d3 you do mention that those are just reference images ... but i am literally just rebuilding the same thing 🙂

Mateusz Gozdek (invidian)
2023-03-01 10:50:55

Yes, CAPI images we publish are reference images for testing and CI use with no regular/security updates guarantees. This is the same for all CAPI images available from the maintainers. So users are recommended to build their own images with versions they need etc.

🙏 fc
fc
2023-03-09 08:20:59

@Mateusz Gozdek (invidian) i feel really bad for asking for this ... but ... any chance you could push 3374.2.5 to flatcar community gallery ? 🙏

Mateusz Gozdek (invidian)
2023-03-09 10:12:09

Sure, it's crunching. I had a day off yesterday 🙂

fc
2023-03-09 10:13:10

🙏 Thanks , very appreciated ... is the last bit i need to check my pipeline will pick it up tonight 🤞

Саша Рудан (Sasha Rudan)
2023-02-24 01:24:11

@Саша Рудан (Sasha Rudan) has joined the channel

Danny Bessems
2023-02-26 09:49:41

@Danny Bessems has joined the channel

Slackbot
2023-02-26 09:49:41

This message was deleted.

Shyam P R
2023-02-27 01:26:49

it may be helpful for folks to know the provider which you are trying

Jhonathan Cavalcante
2023-02-28 02:48:33

@Jhonathan Cavalcante has joined the channel

Kai Lüke
2023-03-01 03:20:08

@Kai Lüke has joined the channel

fc
2023-03-01 09:01:04

Hi Everyone !

I have a question about releases on

the last release is 0.1.12 from May 2022, there was a tag 0.1.13 from Sep 2022 ( which is what i am using for my builds )

but since then a lot of changes have been pushed, including some that are a requirement for recent versions of kubernetes like ( )

Do you think is worth cutting a new release to also limit the amount of changes in between releases ?

Danny Bessems
2023-03-02 08:28:06

Similarly, the latest container image for the image-builder is ancient as well, it is missing all the latest make targets (for instance no Ubuntu 22.04).

fc
2023-03-02 08:30:08

and, i did not check but i hope , that AZ cli will be more up-to-date in the master branch ( or i will do a PR for it ) since a bunch of commands fail with current version 🙂

Danny Bessems
2023-03-02 08:32:45

Do you know what version is in the tarball that is linked in the CAPI Image-builder documentation? Is that then 0.1.12 or does it include all commits since?

fc
2023-03-02 08:33:35

no i don't know, but i don't use 0.1.12 i use the tag 0.1.13 ( which did not get a release though not sure why )

i know flatcar upstream uses master ...

Danny Bessems
2023-03-02 08:34:31

Oh my, that could explain my struggles getting a working Ubuntu OVA. I'll have to check...

🤞 fc
Danny Bessems
2023-03-02 08:38:20

Yeah, the tarball is based on master including all commits (it uses github's tarball api endpoint)

Yike Wang
2023-03-02 11:54:10

so.. will we have a new release recently?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-02 13:28:36

@kiran keshavamurthy @mboersma Do you think we could get a new release cut sometime soon?

kiran keshavamurthy
2023-03-02 18:28:54

Yes, I can create a new tag today or tomorrow. Do we need any open PRs to get in before creating a new tag?

🙏 fc
Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-02 19:55:25

@fc I think you still have one open that would be good to go in don’t you?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-02 19:56:19

Just need an approval https://github.com/kubernetes-sigs/image-builder/pull/1087

GitHub
fc
2023-03-02 19:56:33

Yes , it got assigned and is a very simple one so hopefully it will make it 👍

jsturtevant
2023-03-03 00:00:29

this one too:

GitHub
jsturtevant
2023-03-03 00:00:57

@kiran keshavamurthy Is there docs or maybe we can pair for the release? I don't know that process

kiran keshavamurthy
2023-03-07 01:04:19

@jsturtevant As discussed I’ll update the docs on how to tag and release

:ty: jsturtevant
kiran keshavamurthy
2023-03-07 01:05:33

I’ve created a tag for v0.1.14 today.
@Sanika Gawhane Can you create a PR to promote the container image from staging to prod please?

:ack: Sanika Gawhane, jsturtevant
:ty: jsturtevant
fc
2023-03-07 15:29:05

thanks 👍

Yike Wang
2023-03-02 11:51:17

@Yike Wang has joined the channel

Yike Wang
2023-03-06 15:48:24

Hi everyone, did you ever see such error when make build-ami-flatcar flatcar ami? Any guidance for the checkings? I am using v0.1.13 branch. cc @swan

    amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : unpack crictl] *
amazon-ebs.{{user build&#95;name}}: changed: [default]
amazon-ebs.{{user build&#95;name}}:
amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Remove crictl tarball] *

amazon-ebs.{{user build&#95;name}}: changed: [default]
amazon-ebs.{{user build&#95;name}}:
amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Create kubelet default config file] *
amazon-ebs.{{user build&#95;name}}: changed: [default]
amazon-ebs.{{user build&#95;name}}:
amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Enable kubelet service] *

amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": false, "msg": "Could not find the requested service kubelet: host"}
amazon-ebs.{{user build&#95;name}}:
amazon-ebs.{{user build&#95;name}}: PLAY RECAP **
amazon-ebs.{{user build&#95;name}}: default : ok=45 changed=34 unreachable=0 failed=1 skipped=166 rescued=0 ignored=0
amazon-ebs.{{user build&#95;name}}:

Jeremi Piotrowski
2023-03-06 17:13:06

taking a look

Jeremi Piotrowski
2023-03-06 17:51:01

having trouble running AMI build right now, do you have a full log?

Jeremi Piotrowski
2023-03-06 17:51:20

and have you tried HEAD of the master branch?

Yike Wang
2023-03-07 06:46:52

full log on v0.1.12 (should be same with 0.1.13):

Yike Wang
2023-03-07 06:47:13

*Thread Reply:* None

Yike Wang
2023-03-07 06:49:28

I tried on master or v0.1.14, it is even worse with "ansible-playbook: error: argument --scp-extra-args: expected one argument":

no_proxy=** make build-ami-flatcar
hack/ensure-ansible.sh
Starting galaxy collection install process
....
==> amazon-ebs.{{user build&#95;name}}: Executing Ansible: ansible-playbook -e packer_build_name="flatcar-stable" -e packer_builder_type=amazon-ebs --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url= containerd_sha256=8e227caa318faa136e4387ffd6f96baeaad5582d176202fe9da69cde87036033 pause_image=registry.k8s.io/pause:3.9 containerd_additional_settings= containerd_cri_socket=/var/run/containerd/containerd.sock containerd_version=1.6.8 containerd_wasm_shims_url= containerd_wasm_shims_version=v0.3.3 containerd_wasm_shims_sha256=da84b1c065a58f95a841d39e143cd7115d43e6faedcce7a8782f2942388260d7 containerd_wasm_shims_runtimes="" crictl_url= crictl_sha256= crictl_source_type=http custom_role_names="" firstboot_custom_roles_pre="" firstboot_custom_roles_post="" node_custom_roles_pre="" node_custom_roles_post="" disable_public_repos=false extra_debs="" extra_repos="" extra_rpms="" http_proxy= https_proxy= kubeadm_template=etc/kubeadm.yml kubernetes_cni_http_source= kubernetes_cni_http_checksum=sha256: kubernetes_http_source= kubernetes_container_registry=registry.k8s.io kubernetes_rpm_repo= kubernetes_rpm_gpg_key=" " kubernetes_rpm_gpg_check=True kubernetes_deb_repo=" kubernetes-xenial" kubernetes_deb_gpg_key= kubernetes_cni_deb_version=1.2.0-00 kubernetes_cni_rpm_version=1.2.0-0 kubernetes_cni_semver=v1.2.0 kubernetes_cni_source_type=http kubernetes_semver=v1.26.2 kubernetes_source_type=pkg kubernetes_load_additional_imgs=false kubernetes_deb_version=1.26.2-00 kubernetes_rpm_version=1.26.2-0 no_proxy= pip_conf_file= python_path=/opt/bin/builder-env/site-packages redhat_epel_rpm= epel_rpm_gpg_key= reenable_public_repos=true remove_extra_repos=false systemd_prefix=/etc/systemd sysusr_prefix=/opt sysusrlocal_prefix=/opt load_additional_components=false additional_registry_images=false additional_registry_images_list= additional_url_images=false additional_url_images_list= additional_executables=false additional_executables_list= additional_executables_destination_path= build_target=virt amazon_ssm_agent_rpm= --extra-vars ansible_python_interpreter=/opt/bin/python --extra-vars --scp-extra-args -O -e ansible_ssh_private_key_file=/var/folders/vz/sz23dk7j35x9hkd69x2208s40000gq/T/ansible-key3776119403 -i /var/folders/vz/sz23dk7j35x9hkd69x2208s40000gq/T/packer-provisioner-ansible3049382186 /Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/ansible/node.yml
amazon-ebs.{{user build&#95;name}}: usage: ansible-playbook [-h] [--version] [-v] [-k]
amazon-ebs.{{user build&#95;name}}: [--private-key PRIVATE_KEY_FILE] [-u REMOTE_USER]
amazon-ebs.{{user build&#95;name}}: [-c CONNECTION] [-T TIMEOUT]
amazon-ebs.{{user build&#95;name}}: [--ssh-common-args SSH_COMMON_ARGS]
amazon-ebs.{{user build&#95;name}}: [--sftp-extra-args SFTP_EXTRA_ARGS]
amazon-ebs.{{user build&#95;name}}: [--scp-extra-args SCP_EXTRA_ARGS]
amazon-ebs.{{user build&#95;name}}: [--ssh-extra-args SSH_EXTRA_ARGS] [--force-handlers]
amazon-ebs.{{user build&#95;name}}: [--flush-cache] [-b] [--become-method BECOME_METHOD]
amazon-ebs.{{user build&#95;name}}: [--become-user BECOME_USER] [-K] [-t TAGS]
amazon-ebs.{{user build&#95;name}}: [--skip-tags SKIP_TAGS] [-C] [--syntax-check] [-D]
amazon-ebs.{{user build&#95;name}}: [-i INVENTORY] [--list-hosts] [-l SUBSET]
amazon-ebs.{{user build&#95;name}}: [-e EXTRA_VARS] [--vault-id VAULT_IDS]
amazon-ebs.{{user build&#95;name}}: [--ask-vault-password | --vault-password-file VAULT_PASSWORD_FILES]
amazon-ebs.{{user build&#95;name}}: [-f FORKS] [-M MODULE_PATH] [--list-tasks]
amazon-ebs.{{user build&#95;name}}: [--list-tags] [--step] [--start-at-task START_AT_TASK]
amazon-ebs.{{user build&#95;name}}: playbook [playbook ...]
amazon-ebs.{{user build&#95;name}}: ansible-playbook: error: argument --scp-extra-args: expected one argument
......
==> amazon-ebs.{{user build&#95;name}}: Provisioning step had errors: Running the cleanup provisioner, if present...
==> amazon-ebs.{{user build&#95;name}}: Terminating the source AWS instance...

swan
2023-03-07 06:50:24

ansible error can be addressed by following:

 export ANSIBLE_SCP_EXTRA_ARGS="-O"  
export ANSIBLE_SSH_ARGS="-oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedAlgorithms=+ssh-rsa"

Yike Wang
2023-03-07 07:04:13

I re-read the errors on master, it seems it is just the workaround of export ANSIBLESCPEXTRAARGS="-O" brought to the error:

Executing Ansible: ansible-playbook -e packer_build_name="flatcar-stable" -e packer_builder_type=amazon-ebs --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url= 

ansible-playbook: error: argument --scp-extra-args: expected one argument

So I remove the env ANSIBLE
SCPEXTRAARGS and it can continue now.

👍 swan
Yike Wang
2023-03-07 07:08:59

Still the same thing fo build-ami-flatcar currently on master/0.1.14:

    amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Enable kubelet service] *
amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": false, "msg": "Could not find the requested service kubelet: host"}
amazon-ebs.{{user build&#95;name}}:
amazon-ebs.{{user build&#95;name}}: PLAY RECAP *
**
amazon-ebs.{{user build&#95;name}}: default : ok=49 changed=36 unreachable=0 failed=1 skipped=210 rescued=0 ignored=0

Jeremi Piotrowski
2023-03-07 09:25:25

whats in your config-1.26.2-flatcar.json

Jeremi Piotrowski
2023-03-07 09:25:40

so from your log, you're missing:

Jeremi Piotrowski
2023-03-07 09:25:43
when: kubernetes_source_type == "http" and kubernetes_cni_source_type == "http"
Jeremi Piotrowski
2023-03-07 09:25:49

these are set by default for AMIs

Jeremi Piotrowski
2023-03-07 09:25:53

flatcar AMIs

Jeremi Piotrowski
2023-03-07 09:25:55

but not for you somehow

Yike Wang
2023-03-07 10:39:52

i see! Let me figure out! Thanks!!!

Jeremi Piotrowski
2023-03-07 10:56:46

if you just care about getting an image this should work too:

diff --git a/images/capi/ansible/roles/kubernetes/tasks/main.yml b/images/capi/ansible/roles/kubernetes/tasks/main.yml
index 36d973b39..55885f1ef 100644
--- a/images/capi/ansible/roles/kubernetes/tasks/main.yml
+++ b/images/capi/ansible/roles/kubernetes/tasks/main.yml
@@ -21,6 +21,9 @@
- import_tasks: photon.yml
when: kubernetes_source_type == "pkg" and ansible_os_family == "VMware Photon OS"

+- import_tasks: url.yml
+ when: ansible_os_family == "Flatcar"
+
- name: Symlink cri-tools
file:
src: "/usr/local/bin/{{ item }}"

Yike Wang
2023-03-07 11:30:12

Thanks @Jeremi Piotrowski I added kubernetessourcetype == "http" and kubernetescnisource_type == "http" to my config json and then it works. I wasn't aware of these two things before. Thanks a lot!

Jeremi Piotrowski
2023-03-07 12:16:41

It shouldn’t be needed since it’s part of the flatcar config that is included in before your file

Jeremi Piotrowski
2023-03-07 12:16:51

But something about yours must be interfering

Yike Wang
2023-03-07 13:25:29

I just use this config json but I didn't add below two lines before (not aware of it ..):

cat config-1.26.2-flatcar.json
{
"kubernetes_series": "1.26",
"kubernetes_semver": "v1.26.2",
"kubernetes_rpm_version": "1.26.2-0",
"kubernetes_deb_version": "1.26.2-00",
"kubernetes_source_type": "pkg",
"kubernetes_http_source": "",
"kubernetes_rpm_repo": "",
"kubernetes_rpm_gpg_key": "\" \"",
"kubernetes_rpm_gpg_check": "True",
"kubernetes_deb_repo": "\" kubernetes-xenial\"",
"kubernetes_deb_gpg_key": "",
"kubernetes_container_registry": "registry.k8s.io",
"kubernetes_load_additional_imgs": "false",
"kubeadm_template": "etc/kubeadm.yml",
"containerd_version": "1.6.8",
"containerd_sha256": "8e227caa318faa136e4387ffd6f96baeaad5582d176202fe9da69cde87036033",
#"kubernetes_source_type": "http",
#"kubernetes_cni_source_type": "http"
}

Varun Kumar Vellanki
2023-03-06 17:48:58

@Varun Kumar Vellanki has joined the channel

Jhonathan Cavalcante
2023-03-06 22:27:31

[openstack Built CAPI Openstack Image breaking in kubeadm init command]

Jhonathan Cavalcante
2023-03-06 22:27:35

I'm building a kubernetes image for openstack. The build processes is running ok, the image is built and I'm able to provision a instance with it. The problem is that apparently cloud-init is running a kubeadm init command with the wrong kubeadm.yaml config, it looks like it is over righting the config with a file in /run/kubeadm/kubeadm.yaml. This is causing coredns pull to fail and break the provisioning process.

I've changed the registry value in the following files:

cloudinit/user-data:      imageRepository: registry.k8s.io
packer/config/kubernetes.json: "kubernetes_container_registry": "registry.k8s.io",
packer/qemu/packer.json: "kubernetes_container_registry": "registry.k8s.io",
packer_kubernetes.json: "kubernetes_container_registry": "registry.k8s.io",

Not sure how cloud-init works on spining up the vms when I ask for a new instance, maybe there's some init bootstrap config to be made.

I've started to have this issue with k8s 1.24.10, the image built works well with k8s 1.22.9. I've also noticed some issues around the k8s registry, but nothing about this particular issue while provisioning the instance. I'd be glad if anyone can help me with this. Thanks!

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-07 08:00:27

Which version of kube-builder are you using? This should have been resolved with

GitHub
Jhonathan Cavalcante
2023-03-07 13:23:35

I'm using the latest release, just check now there's some tags with other releases, I'll be updating with all the changes. But we already had this configs inside the repo.

The thing is that cloud-init is apparently running a kubeadm init command that overrides this configuration from user-data.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-07 13:27:52

Is it that the kubeadm version being used is too old? Which provider is used for Openstack in image-builder?

Jhonathan Cavalcante
2023-03-07 17:43:37

I'm using qemu in ubuntu 20.04

Jhonathan Cavalcante
2023-03-07 17:55:49

just built from the last tag 0.1.14, and got the same error. it seems that kubeadm is writing its own kubeadm config file

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-09 12:38:49

Sorry, just got chance to look at this. It seems that the containerd_version isn't specified for that target so it relies on whatever is installed from Ubuntu 20.04 I think. I guess we need to specify the specific version to use. I'm not sure what version exactly is best.

Jhonathan Cavalcante
2023-03-09 12:40:57

Some updates on this. CAPI was overwriting the container registry address, so It was a issue on my side. But the 1.26.1 version of crictl is breaking a 1.22.9 k8s image for example. I think that can be related to containerd_version.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-09 12:43:03

Possibly, I get quite confused with all the inter-dependancies 😅

Jhonathan Cavalcante
2023-03-09 12:43:24

It is quite complex to understand

Sarva Bhowma
2023-03-08 20:56:08

@Sarva Bhowma has joined the channel

Abhay Krishna Arunachalam
2023-03-09 19:43:03

Hello Image-builder maintainers, I have a question. The packer.json for AMI builds has a throughput field by default in the block device mappings section with a default value of 125, but this field is valid only for gp3 volumes. I tried building a gp2 Ubuntu AMI by overriding the throughput from a different JSON var-file by setting it to "" or even null , but in both cases the build failed saying

Throughput is not available for device /dev/sda1
I was able to build it only after removing the throughput field from the packer.json. But removal also doesnt seem to be a solution because now there's no way of passing throughput in if I do want to build a GP3 volume. AMI missing something (pun intended 😛)? Is this a bug?

Abhay Krishna Arunachalam
2023-03-10 04:39:13

Created an issue for this:

GitHub
👍 jsturtevant
Sanika Gawhane
2023-03-10 22:23:21

Hello everyone, @kiran keshavamurthy and I are trying to bump the upstream image-builder to the latest tag/release v0.1.14 . Kiran created the tag and release successfully. However, the container build that was automatically kicked off pushed the container to the registry without a tag.
Are we missing something here? Does anything in cloudbuild.yaml need to be modified while creating the tag/release?

GitHub
Abhay Krishna Arunachalam
2023-03-13 21:16:27

@kiran keshavamurthy what do you think of these changes?
Background: We were hit by a Packer build issue where the incorrect packer binary was being used. We were building on a RHEL image, which already had a system-level package providing a packer binary, which conflicted with Hashicorp's Packer

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-14 15:51:13

Hey y'all 👋 are any of the other contributors and maintainers of image-builder going to be at Kubecon next month in Amsterdam? I'd be interested in a meetup to discuss the project, pain points, future, etc. if anyone was keen. Maybe at the contributor summit? (/cc @kiran keshavamurthy @jsturtevant @mboersma)

jsturtevant
2023-03-14 16:16:15

I won't be there this time

😞 Marcus Noble
mboersma
2023-03-14 19:41:39

I won't either, but I very much endorse having a summit meeting about image-builder and establishing some future directions, etc.

💙 Marcus Noble, Julien Klaer
kiran keshavamurthy
2023-03-16 19:07:46

I won’t be there either.

kiran keshavamurthy
2023-03-16 19:08:03

I think a restart of the office hours would be useful.

👍 mboersma, knfoo, jsturtevant, Marcus Noble
jsturtevant
2023-03-17 15:24:32

agreed, we have had quite a bit of activity in the last few months.

:nod: Marcus Noble
jsturtevant
2023-03-17 15:49:05

In other sigs we used a doodle to collect times, or we can set one that works for the majority of the maintainers. thoughts?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-17 16:01:08

I think a doodle would be useful. I think we have quite a split between US and EU timezones so I think it might be hard to find a time that pleases everyone.

👍 jsturtevant
jsturtevant
2023-03-17 16:16:46

any interest in taking lead on that?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-17 16:23:05

Sure, but I'll pick it up on Monday. 🙂

:ty: jsturtevant
Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-20 09:23:09

Wow it's been a long time since I've used Doodle. It's got so many ads these days 🙈

Anyway, before I share into the main channel I'd like to get your thoughts first so here it is:

I went with constraints of no early than 8am PDT and no later than 7pm GMT. Hopefully that's good enough for most people.

doodle.com
👍 jsturtevant
jsturtevant
2023-03-20 16:03:27

@kiran keshavamurthy @mboersma ^^^

kiran keshavamurthy
2023-03-20 17:16:55

Thanks for setting it up Marcus. I’ve submitted my times.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-20 17:19:59

Cool. I'll post it into the channel then. I'm thinking deadline for the end of this month to give people some time to answer but not have it drag on too long.

jsturtevant
2023-03-20 18:11:31

The days that have times today are grey out since they are in the past 🤨. Might need to put the dates in April

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-20 18:28:59

🤦‍♂️ I'll try and update it. It might cause the answers to need re-doing.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-20 18:29:56

Actually, I'm just going to create a new one and update the link. I'll let you know when its up so you can answer again. Sorry about that.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-20 18:35:08

Right, updated the link to
@jsturtevant @kiran keshavamurthy you'll need to fill it out again.

:done_2: kiran keshavamurthy, jsturtevant, mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-28 08:52:44

Just a reminder that we're looking for peoples thoughts on the best day/time to run the office hours. Could all those who might be interested in attending at some point please fill out the above Doodle poll with your favoured slot. 🙏

👍 mboersma, jsturtevant
Marcus Noble (k8s@marcusnoble.co.uk)
2023-03-30 15:19:04

Just one last reminder asking for peoples thoughts on this. 🙂 I'm going to close the poll tomorrow afternoon (UTC).
So far, Monday at 3:30pm UTC looks to be the favoured new slot for the office hours.

👍 jsturtevant, mboersma, kiran keshavamurthy
pengzhan
2023-03-21 03:45:52

@pengzhan has joined the channel

Abhishek Jha
2023-03-22 02:07:00

@Abhishek Jha has joined the channel

Maximilian Rink
2023-03-23 18:07:46

hey folks,
we are running into some issues with the ubuntu2204 images, namely with cloud-init. Maybe someone has seen it before 😅

Maximilian Rink
2023-03-23 18:08:17

That error isnt present at every start, but on about 70% of them

Drew Hudson-Viles
2023-03-27 14:05:07

Can you log into the node and check the cloud-init logs in /var/log?

I'm curious to know what you're seeing in there as I've ran into an issue in the past few days since I've synced with upstream and it looks like cloud-init might be failing. It may be purely coincidental though and I'm not sure if it's a problem with image builder as I've tested using an image that was built before I started having this issue.
It was previously working and is currently not making me wonder if this could be a problem with CAPI itself. I'm still looking into this on my side.

Drew Hudson-Viles
2023-03-28 16:52:21

I forgot to update you here. It was just a configuration issue on my side causing the cloud-init issue picard_facepalm. So it's very likely I can't assist you with my fix. However I do recommend if it's failing to check the cloud init logs to see what's going on in there.

aniruddha
2023-03-27 17:08:33

@aniruddha has joined the channel

aniruddha
2023-03-27 17:09:02



Need some final review on this

cc @richcase @kiran keshavamurthy

Abhay Krishna Arunachalam
2023-03-29 00:25:38

@kiran keshavamurthy Thoughts on this?

This is a duplicate of another PR that i had opened earlier and closed (#870)

GitHub
knfoo
2023-03-29 15:13:31

Hi 👋
I am preparing a PR on ubuntu-22.04 efi on qemu which requires a user-data file that is different from the file that is in ova - can you advise on where to put that file / name it differently like prefix it with qemu-

knfoo
2023-03-29 15:16:55

Or should this question be discussed in the PR ?

Drew Hudson-Viles
2023-03-29 20:05:15

Hello!

I'd say stick the PR in as a draft and then have the discussion in there. It'll give a "paper trail" for anyone else who may need to do something similar in future and helps any reviewers make decisions based on the conversation that happens in there. I recently had a pretty large PR go in that had something like 94 entries in the conversation tab (minus a few bot entries) but it was great to have it in there as anyone coming into the conversation had the history.

knfoo
2023-03-30 20:03:49

Great I will do that 👍

Marcus Noble (k8s@marcusnoble.co.uk)
2023-04-03 07:13:42

Right, closing the above poll. Based on the votes it looks like Mondays at 3:30pm UTC is the most favoured time slot for the office hours!

🎉 bavarianbidi, Drew Hudson-Viles, mboersma, jsturtevant
Drew Hudson-Viles
2023-04-04 12:15:29

Do you have a date for when the the first one will take place? (I only ask as the link takes me to March 30th 😉 )

Will you also drop a link in here for access to the meeting? It's going to be a first for me so not sure where to go for any of it 🙂

Thanks!

Marcus Noble (k8s@marcusnoble.co.uk)
2023-04-04 12:18:27

Unsure right now. We wanted to find a suitable slot first.

Drew Hudson-Viles
2023-04-04 13:36:28

No problem 🙂 I'm getting ahead of myself!

mboersma
2023-04-05 17:49:15

Next Monday, April 10? I can be there.

👍 Drew Hudson-Viles, jsturtevant
jsturtevant
2023-04-05 21:43:42

We can get the readme updated: .

GitHub
jsturtevant
2023-04-05 21:44:18

I not sure who has access to update the calendar and zoom meeting time? @kiran keshavamurthy?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-04-06 09:33:36

The next two Mondays are no good for me (UK bank holiday then Kubecon) but would be happy for it to go ahead without me 🙂

Drew Hudson-Viles
2023-04-06 09:37:46

oh that's a good point - both of those things are applicable to me too. Baby brain is a killer.

😅 Marcus Noble
mboersma
2023-04-06 16:10:49

We can always wait a week or two if that's better timing. First we should sort out the calendar so the new time is publicly visible.

mboersma
2023-04-06 16:27:43

I created the kubernetes/community PR to change the meeting time here:
(However, when I changed the time previously for CAPZ office hours, this wasn't sufficient to actually make the change on the public calendar, we may have to nudge someone...)

GitHub
✅ Marcus Noble
:thx_thanks: Marcus Noble, kiran keshavamurthy
jsturtevant
2023-04-19 19:23:22

I've updated here too:

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2023-04-24 12:50:37

How do we go about getting this added to the public calendar?

mboersma
2023-04-24 16:19:14

I’m not sure. I thought the kubernetes/community PR would do it.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-04-24 16:21:31

Doesn't look like it I don't think. Unless my GCal isn't updating it? thinking

Marcus Noble (k8s@marcusnoble.co.uk)
2023-04-24 16:23:43

Yeah, not showing there:

Google Workspace
Marcus Noble (k8s@marcusnoble.co.uk)
2023-04-24 16:26:19

This seems to suggest a specific calendar needs to be created and shared. Not sure if that's still correct or not -

Kubernetes Contributors
👍 mboersma
Nitish Chauhan
2023-04-04 05:50:33

@Nitish Chauhan has joined the channel

Praveen Adini
2023-04-17 21:48:25

@Praveen Adini has joined the channel

sirhopcount
2023-04-18 09:02:49

@sirhopcount has joined the channel

Praveen Adini
2023-04-18 15:46:48

Hi Team, I built a RHEL 8 worker node AMI using image-builder and when I manually created a nodegroup for an existing EKS using an ASG, the nodes didn't get attached to the cluster. Upon further investigation I see some of the necessary scripts are missing from the image like /etc/eks/bootstrap.sh, etc. Wondering if its possible to build a worker node AMI using image-builder and if so, are they any further customizations that I have to perform.

voor
2023-04-18 20:11:22

Historically image builder was designed for kubeadm based clusters, of which EKS (to the best of my knowledge) is not one, I would recommend looking at

GitHub
Praveen Adini
2023-04-18 20:24:18

thanks @voor

rodrigodelmonte
2023-04-20 10:21:02

@rodrigodelmonte has joined the channel

fc
2023-04-20 14:32:38

Hi, we build our capz flatcar images from the flatcar community library as a base but the 3510.2.0 is not available yet

pinging @Mateusz Gozdek (invidian) for reference ( since in the past you triggered the push )

➜ az sig image-version list-community --location westeurope --public-gallery-name flatcar-23485951-527a-48d6-9d11-6931ff0afc2e --only-show-errors --gallery-image-definition flatcar-stable-amd64 -o json | jq '.[]
.name'
"3374.2.0"
"3374.2.1"
"3374.2.3"
"3374.2.4"
"3374.2.5"

l4z41
2023-04-22 10:11:09

@l4z41 has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2023-04-24 15:57:45

Hey y'all 👋, the image-builder office hours will be taking place in roughly 30 minutes at half past.
You can find the meeting notes here:

🙌 Drew Hudson-Viles, mboersma, jsturtevant
jsturtevant
2023-04-24 19:59:31

Looking for some feedback on removing some of the projects which are not actively maintained:

:thx_thanks: Marcus Noble
👍 Drew Hudson-Viles
🧹 mboersma
Drew Hudson-Viles
2023-04-24 22:33:02

@Marcus Noble @jsturtevant @mboersma
Thought you may be interested in this based on the earlier discussion during office hours:

docs.nvidia.com
👍 Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2023-04-25 11:08:32

@mboersma I just noticed that the image-builder event is now correct in the Kubernetes calendar! 😄 Was that your doing?

mboersma
2023-04-25 17:36:45

@cecile is an SCL lead, she fixed it. 🙂

🎉 Marcus Noble
Mike Resvanis
2023-04-26 13:36:35

@Mike Resvanis has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2023-05-02 09:41:23

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 2:30PM every other Monday (next occurrence is May 8th), Greenwich Mean Time.

:ty: jsturtevant, mboersma
mloskot
2023-05-04 09:58:39

@mloskot has joined the channel

mloskot
2023-05-04 10:02:56

Hi, I'm new here, so hello everyone. As part of my activities around the sig-windows-dev-tools, I'm happy to share that after several days of attempts I've completed my first successful, I think, run of the image-builder on Windows generating Windows Server image running the builder from WSL. I do realise WSL is not a typical environment for image-builder users, but I just wanted to share that it is feasible to use WSL.

Mateusz Loskot (https://kubernetes.slack.com/team/U04R6C6HHC5)
🎉 jsturtevant
Christophe Jauffret
2023-05-04 13:27:25

@Christophe Jauffret has joined the channel

Yike Wang
2023-05-05 11:14:37

Hi folks, when I use v0.1.14 tag branch to build ami, I meet an issue:

amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": true, "cmd": "kubeadm config images pull --config /etc/kubeadm.yml --cri-socket /var/run/containerd/containerd.sock", "delta": "0:00:00.036572", "end": "2023-05-05 03:06:53.596128", "msg": "non-zero return code", "rc": 1, "start": "2023-05-05 03:06:53.559556", "stderr": "your configuration file uses an old API spec: \"kubeadm.k8s.io/v1beta2\". Please use kubeadm v1.22 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["your configuration file uses an old API spec: \"kubeadm.k8s.io/v1beta2\". Please use kubeadm v1.22 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "", "stdout_lines": []}
any idea? Thanks!

Yike Wang
2023-05-05 11:16:24

Full log:

no_proxy=** make build-ami-ubuntu-2004
hack/ensure-ansible.sh
Starting galaxy collection install process
Nothing to do. All requested collections are already installed. If you want to reinstall them, consider using --force.
hack/ensure-ansible-windows.sh
IMPORTANT: Winrm connection plugin for Ansible on MacOS causes connection issues.
See for more details.
To fix the issue provide the enviroment variable 'no_proxy='
Example call to build Windows images on MacOS: 'no_proxy=
make build-'
hack/ensure-packer.sh
hack/ensure-goss.sh
Right version of binary present
packer build -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/kubernetes.json" -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/cni.json" -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/containerd.json" -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/wasm-shims.json" -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/ansible-args.json" -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/goss-args.json" -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/common.json" -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/config/additional_components.json" -color=true -var-file="/Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/packer/ami/ubuntu-2004.json" -var-file="/Users/yikew/Working/capa/image-builder/1.27/config.json" packer/ami/packer.json
amazon-ebs.{{user build&#95;name}}: output will be in this color.

Yike Wang
2023-05-05 11:16:42
==> amazon-ebs.{{user `build_name`}}: Prevalidating any provided VPC information
==> amazon-ebs.{{user `build_name`}}: Prevalidating AMI Name: capa-ami-ubuntu-20.04-v1.27.0-1683255429
amazon-ebs.{{user `build_name`}}: Found Image ID: ami-0481e8ba7f486bd99
==> amazon-ebs.{{user `build_name`}}: Creating temporary keypair: packer_64547086-412f-e577-58a5-b431924ffd0d
==> amazon-ebs.{{user `build_name`}}: Creating temporary security group for this instance: packer_6454708e-4b3f-2bec-e059-ec5d99783a1b
==> amazon-ebs.{{user `build_name`}}: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebs.{{user `build_name`}}: Launching a source AWS instance...
amazon-ebs.{{user `build_name`}}: Instance ID: i-00e09efbb79654a7b
==> amazon-ebs.{{user `build_name`}}: Waiting for instance (i-00e09efbb79654a7b) to become ready...
==> amazon-ebs.{{user `build_name`}}: Using SSH communicator to connect: 44.202.183.215
==> amazon-ebs.{{user `build_name`}}: Waiting for SSH to become available...
==> amazon-ebs.{{user `build_name`}}: Connected to SSH!
==> amazon-ebs.{{user `build_name`}}: Provisioning with shell script: /var/folders/vz/sz23dk7j35x9hkd69x2208s40000gq/T/packer-shell604033847
==> amazon-ebs.{{user `build_name`}}: Provisioning with shell script: ./packer/files/flatcar/scripts/bootstrap-flatcar.sh
==> amazon-ebs.{{user `build_name`}}: Provisioning with Ansible...
amazon-ebs.{{user `build_name`}}: Setting up proxy adapter for Ansible....
==> amazon-ebs.{{user `build_name`}}: Executing Ansible: ansible-playbook -e packer_build_name="ubuntu-20.04" -e packer_builder_type=amazon-ebs --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url= containerd_sha256=1d86b534c7bba51b78a7eeb1b67dd2ac6c0edeb01c034cc5f590d5ccd824b416 pause_image=registry.k8s.io/pause:3.9 containerd_additional_settings= containerd_cri_socket=/var/run/containerd/containerd.sock containerd_version=1.6.20 containerd_wasm_shims_url= containerd_wasm_shims_version=v0.3.3 containerd_wasm_shims_sha256=da84b1c065a58f95a841d39e143cd7115d43e6faedcce7a8782f2942388260d7 containerd_wasm_shims_runtimes="" crictl_url= crictl_sha256= crictl_source_type=pkg custom_role_names="" firstboot_custom_roles_pre="" firstboot_custom_roles_post="" node_custom_roles_pre="" node_custom_roles_post="" disable_public_repos=false extra_debs="" extra_repos="" extra_rpms="" http_proxy= https_proxy= kubeadm_template=etc/kubeadm.yml kubernetes_cni_http_source= kubernetes_cni_http_checksum=sha256: kubernetes_http_source= kubernetes_container_registry=registry.k8s.io kubernetes_rpm_repo= kubernetes_rpm_gpg_key=" " kubernetes_rpm_gpg_check=True kubernetes_deb_repo=" kubernetes-xenial" kubernetes_deb_gpg_key= kubernetes_cni_deb_version=1.2.0-00 kubernetes_cni_rpm_version=1.2.0-0 kubernetes_cni_semver=v1.2.0 kubernetes_cni_source_type=pkg kubernetes_semver=v1.27.0 kubernetes_source_type=pkg kubernetes_load_additional_imgs=false kubernetes_deb_version=1.27.0-00 kubernetes_rpm_version=1.27.0-0 no_proxy= pip_conf_file= python_path= redhat_epel_rpm= epel_rpm_gpg_key= reenable_public_repos=true remove_extra_repos=false systemd_prefix=/usr/lib/systemd sysusr_prefix=/usr sysusrlocal_prefix=/usr/local load_additional_components=false additional_registry_images=false additional_registry_images_list= additional_url_images=false additional_url_images_list= additional_executables=false additional_executables_list= additional_executables_destination_path= build_target=virt amazon_ssm_agent_rpm= --extra-vars --extra-vars --scp-extra-args "-O" -e ansible_ssh_private_key_file=/var/folders/vz/sz23dk7j35x9hkd69x2208s40000gq/T/ansible-key68078117 -i /var/folders/vz/sz23dk7j35x9hkd69x2208s40000gq/T/packer-provisioner-ansible1682039367 /Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/ansible/node.yml
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: PLAY [all] ******************************************************************************************************************************************
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [Gathering Facts] ******************************************************************************************************************
amazon-ebs.{{user `build_name`}}: ok: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [include_role : node] **********************************************************************************************************
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [setup : Put templated sources.list in place] **********************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [setup : Put templated apt.conf.d/90proxy in place when defined] ********************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [setup : perform a dist-upgrade] ************************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [setup : install baseline dependencies] **********************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [setup : install extra debs] ********************************************************************************************
amazon-ebs.{{user `build_name`}}: ok: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [setup : install pinned debs] ******************************************************************************************
amazon-ebs.{{user `build_name`}}: ok: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [node : Ensure overlay module is present] ******************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [node : Ensure br_netfilter module is present] ********************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [node : Persist required kernel modules] ********************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [node : Set and persist kernel params] ************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.bridge.bridge-nf-call-iptables', 'val': 1})
amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.bridge.bridge-nf-call-ip6tables', 'val': 1})
amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.ipv4.ip_forward', 'val': 1})
amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.ipv6.conf.all.forwarding', 'val': 1})
amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.ipv6.conf.all.disable_ipv6', 'val': 0})
amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'net.ipv4.tcp_congestion_control', 'val': 'bbr'})
amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'vm.overcommit_memory', 'val': 1})
amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'kernel.panic', 'val': 10})
amazon-ebs.{{user `build_name`}}: changed: [default] => (item={'param': 'kernel.panic_on_oops', 'val': 1})
Yike Wang
2023-05-05 11:16:58
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [node : Ensure auditd is running and comes on at reboot] ************************************
amazon-ebs.{{user `build_name`}}: ok: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [node : configure auditd rules for containerd] ********************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [node : Ensure reverse packet filtering is set as strict] **********************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [node : Copy udev etcd network tuning rules] ************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [node : Copy etcd network tuning script] ********************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [include_role : providers] ************************************************************************************************
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : include_tasks] **********************************************************************************************
amazon-ebs.{{user `build_name`}}: included: /Users/yikew/Projects/src/github.com/kubernetes-sigs/image-builder/images/capi/ansible/roles/providers/tasks/aws.yml for default
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : upgrade pip to latest] ******************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : install aws clients] **********************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : install aws agents Ubuntu] **********************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : Ensure ssm agent is running Ubuntu] ****************************************************
amazon-ebs.{{user `build_name`}}: ok: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : Disable Hyper-V KVP protocol daemon on Ubuntu] ******************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : Creates unit file directory for cloud-final] **********************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : Create cloud-final boot order drop in file] ************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : Creates unit file directory for cloud-config] ********************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : Create cloud-final boot order drop in file] ************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : Make sure all cloud init services are enabled] ******************************
amazon-ebs.{{user `build_name`}}: ok: [default] => (item=cloud-final)
amazon-ebs.{{user `build_name`}}: ok: [default] => (item=cloud-config)
amazon-ebs.{{user `build_name`}}: ok: [default] => (item=cloud-init)
amazon-ebs.{{user `build_name`}}: ok: [default] => (item=cloud-init-local)
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : Create cloud-init config file] **************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : set cloudinit feature flags] ******************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [providers : Ensure chrony is running] ************************************************************************
amazon-ebs.{{user `build_name`}}: ok: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [include_role : containerd] **********************************************************************************************
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : Install libseccomp2 package] ****************************************************************
amazon-ebs.{{user `build_name`}}: ok: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : download containerd] ********************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : Create a directory if it does not exist] ****************************************
amazon-ebs.{{user `build_name`}}: ok: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : unpack containerd] ************************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : delete /opt/cni directory] ********************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : delete /etc/cni directory] ********************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : Creates unit file directory] ****************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : Create containerd memory pressure drop in file] **************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : Create containerd max tasks drop in file] **************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : Create containerd http proxy conf file if needed] **********************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : Creates containerd config directory] ************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : Copy in containerd config file etc/containerd/config.toml] ******
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : Copy in crictl config] ****************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : start containerd service] **********************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : delete tarball] ******************************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [containerd : delete tarball] ******************************************************************************************
amazon-ebs.{{user `build_name`}}: ok: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [include_role : kubernetes] **********************************************************************************************
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Add the Kubernetes repo key] ****************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Add the Kubernetes repo] ************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Install Kubernetes] **********************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Symlink cri-tools] ************************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default] => (item=ctr)
amazon-ebs.{{user `build_name`}}: changed: [default] => (item=crictl)
amazon-ebs.{{user `build_name`}}: changed: [default] => (item=critest)
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Create kubelet default config file] **************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Enable kubelet service] **************************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Create the Kubernetes version file] **************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Check if Kubernetes container registry is using Amazon ECR] ******
amazon-ebs.{{user `build_name`}}: ok: [default]
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Create kubeadm config file] ******************************************************************
amazon-ebs.{{user `build_name`}}: changed: [default]
amazon-ebs.{{user `build_name`}}:
Yike Wang
2023-05-05 11:17:03
amazon-ebs.{{user `build_name`}}: TASK [kubernetes : Kubeadm pull images] ********************************************************************************
amazon-ebs.{{user `build_name`}}: fatal: [default]: FAILED! => {"changed": true, "cmd": "kubeadm config images pull --config /etc/kubeadm.yml --cri-socket /var/run/containerd/containerd.sock", "delta": "0:00:00.036572", "end": "2023-05-05 03:06:53.596128", "msg": "non-zero return code", "rc": 1, "start": "2023-05-05 03:06:53.559556", "stderr": "your configuration file uses an old API spec: \"kubeadm.k8s.io/v1beta2\". Please use kubeadm v1.22 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["your configuration file uses an old API spec: \"kubeadm.k8s.io/v1beta2\". Please use kubeadm v1.22 instead and run 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "", "stdout_lines": []}
amazon-ebs.{{user `build_name`}}:
amazon-ebs.{{user `build_name`}}: PLAY RECAP ******************************************************************************************************************************************
amazon-ebs.{{user `build_name`}}: default : ok=55 changed=43 unreachable=0 failed=1 skipped=205 rescued=0 ignored=0
amazon-ebs.{{user `build_name`}}:
==> amazon-ebs.{{user `build_name`}}: Provisioning step had errors: Running the cleanup provisioner, if present...
==> amazon-ebs.{{user `build_name`}}: Terminating the source AWS instance...
==> amazon-ebs.{{user `build_name`}}: Cleaning up any extra volumes...
==> amazon-ebs.{{user `build_name`}}: No volumes to clean up, skipping
==> amazon-ebs.{{user `build_name`}}: Deleting temporary security group...
==> amazon-ebs.{{user `build_name`}}: Deleting temporary keypair...
Build 'amazon-ebs.{{user `build_name`}}' errored after 10 minutes 36 seconds: Error executing Ansible: Non-zero exit status: exit status 2

==> Wait completed after 10 minutes 36 seconds

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs.{{user `build_name`}}: Error executing Ansible: Non-zero exit status: exit status 2

==> Builds finished but no artifacts were created.
Yike Wang
2023-05-05 11:17:53

my config file:

{
"kubernetes_series": "1.27",
"kubernetes_semver": "v1.27.0",
"kubernetes_rpm_version": "1.27.0-0",
"kubernetes_deb_version": "1.27.0-00",
"kubernetes_source_type": "pkg",
"kubernetes_http_source": "",
"kubernetes_rpm_repo": "",
"kubernetes_rpm_gpg_key": "\" \"",
"kubernetes_rpm_gpg_check": "True",
"kubernetes_deb_repo": "\" kubernetes-xenial\"",
"kubernetes_deb_gpg_key": "",
"kubernetes_container_registry": "registry.k8s.io",
"kubernetes_load_additional_imgs": "false",
"kubeadm_template": "etc/kubeadm.yml",
"containerd_version": "1.6.20",
"containerd_sha256": "1d86b534c7bba51b78a7eeb1b67dd2ac6c0edeb01c034cc5f590d5ccd824b416"
}

Marcus Noble (k8s@marcusnoble.co.uk)
2023-05-05 11:29:03

The error seems to suggest that you're using a now deprecated version of the kubeadm configfile.
This was resolved a couple weeks ago in but hasn't made it into a release yet. You'll be able to get around it for now by using the latest from the master branch until a new release is released.
@mboersma @jsturtevant we spoke about releases in the last office hours, looks like we could do with doing one so that we support Kubernetes v1.27 with a tagged release of image-builder.

GitHub
👏 Yike Wang
👍 Yike Wang
:thanks: Yike Wang
:1000000: Yike Wang
Yike Wang
2023-05-05 11:35:18

ok then let me use master branch to build 1.27 related amis. Thanks a lot!

jsturtevant
2023-05-11 21:29:35

we have a release out:

Matt Boersma (https://kubernetes.slack.com/team/U5PL62ULA)
👏 Yike Wang
Slackbot
2023-05-08 15:30:08

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma
Drew Hudson-Viles
2023-05-08 15:31:31

I may not make this office hours due to moving house and everything being all over the place in the new house right now. I will try though.

mboersma
2023-05-08 16:44:13

We need to do a fresh release of image-builder, but there is some confusion about the process and the last attempt apparently didn't publish a tagged image.

mboersma
2023-05-08 16:46:17

@kiran keshavamurthy @jsturtevant @Marcus Noble should we get together and see if we can sort things out? (Or does anyone already have a handle on things?)

:yes: Marcus Noble, jsturtevant
jsturtevant
2023-05-08 17:13:16

Yes, Sorry I missed the meeting today. I didn't have it on the calendar. I can do something later in the week?

👍 mboersma
kiran keshavamurthy
2023-05-08 23:46:41

Yeah I’m ok meeting sometime this week to figure it out.

mloskot
2023-05-09 13:11:50

confusion about the process
Is there anything in it related to the Windows images?

jsturtevant
2023-05-09 23:39:33

@mloskot not sure I follow your question. This for the release of the image-builder Docker image and tagging the repository. The docker image is Linux only but can get used with various providers to create Windows. Does that help?

jsturtevant
2023-05-09 23:40:32

folks up for doing a session on Thursday? Maybe 8:30 pacific so European timezone can join if they want?

👍 Marcus Noble, mboersma, kiran keshavamurthy
mloskot
2023-05-10 08:27:16

@jsturtevant Yes, I realise it's not very concrete. I was trying to make myself aware of image-builder issues w.r.t. Windows.

👍 jsturtevant
jsturtevant
2023-05-10 17:36:51

I don't know If I can create a meeting invite but we can re-use the link to the weekly meeting. We can all just sign on at that time.

mboersma
2023-05-10 17:56:29

Works for me, let's try for tomorrow a.m.

👍 jsturtevant
Marcus Noble (k8s@marcusnoble.co.uk)
2023-05-10 18:16:47

I’ll try and attend but might be late.

👍 jsturtevant
jsturtevant
2023-05-10 19:04:28

@kiran keshavamurthy does that work for you? I think you have the most context but I am sure we can figure it out 🙂

kiran keshavamurthy
2023-05-10 19:05:44

I think so. I have some home repair guys coming in between 8-10am. But I should be able to be on the call and chime in.

:ty: jsturtevant
mboersma
2023-05-11 16:29:41

Meeting to share knowledge about image-builder publishing process starting now at , please join if you're interested and available!

:ty: jsturtevant
✅ mloskot
jsturtevant
2023-05-11 16:30:18

I was just about to do this 🙂

jsturtevant
2023-05-11 17:28:22

We have a PR for promotion:

GitHub
jsturtevant
2023-05-11 17:28:26

thanks all!

Marcus Noble (k8s@marcusnoble.co.uk)
2023-05-11 17:28:59

Just found another location we need to update owners:

GitHub
👍 jsturtevant, mboersma
jsturtevant
2023-05-11 18:40:59

@kiran keshavamurthy We should get you added to the k8s org 🙂

GitHub
mboersma
2023-05-11 20:59:02

Image-builder v0.1.15 is now available:
Thanks to all the (many) contributors! thank_you

:nice: mloskot, Jon Zeolla
:wookie_party_time: Joe Kratzat
:parrotk8s: Marcus Noble
:yay2: knfoo
mboersma
2023-05-11 21:01:45

I've started a PR to add some Makefile stuff and docs for releasing, just FYI so we don't duplicate effort.

:ty: jsturtevant, Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2023-05-12 07:32:58

I just noticed that the next date in the office hours notes is down as the 15th but the next in the calendar is the 22nd. Am I correct that its a mistake in the notes or are we wanting to meet up early?

jsturtevant
2023-05-12 16:47:27

I believe the calendar is correct as we had one this week

mboersma
2023-05-12 16:53:06

That was my mistake, date math is hard. I updated the notes to the 22nd.

😃 jsturtevant
Marcus Noble (k8s@marcusnoble.co.uk)
2023-05-12 16:53:21

👍 Just wanted to be sure 🙂

mloskot
2023-05-13 11:08:34

Has anyone tried running the image builder with VirtualBox lately?
After pretty good experiments on Windows host (with WSL), I'm now trying the IB's canonical way, that is on Linux machine with Ubuntu 22.04 and I'm getting errors, but before I start spamming GitHub with issues and PRs, I'd like confirm if IB workflow w/ VirtualBox is still sounds or needs updating indeed.

jsturtevant
2023-05-16 17:30:42

are you building linux images on wsl? I haven't used VB provider from wsl

mloskot
2023-05-16 17:36:17

@jsturtevant
Yes, I have experimented with it, but it requires a hybrid environment: WSL is only used as proxy running Ansible, Packer and Vagrant, but VirtualBox runs on Windows host
Here is my branch with single commit with scratchnotes and changes I had to apply as well as full log


However, I've ditched this idea as there seem to be too much gymnastics needed, and hardcoded IPs, and I don't see how this could be made into generic changes approvable by image builder and SWDT.
Instead, I've got a Linux machine where I'm going to try building images

👍 jsturtevant
Xavier Serrat
2023-05-13 22:10:07

@Xavier Serrat has joined the channel

👋 jsturtevant
Daniel Mello Urbano
2023-05-16 13:20:59

@Daniel Mello Urbano has joined the channel

Danny Bessems
2023-05-22 12:57:18

I'm experiencing a weird error when building a RHEL8 ova for CAPV; after defining values in additional_components.json just like I've done for Ubuntu18/20/22:

  "additional_registry_images": "true",
"additional_registry_images_list": "ghcr.io/kube-vip/kube-vip:v0.5.5",
I see the following error when Packer starts the Ansible provisioner:
    vsphere-iso.vsphere: TASK [Gathering Facts] **
vsphere-iso.vsphere: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: command-line line 0: garbage at end of line; \"-o\".", "unreachable": true}

Danny Bessems
2023-05-22 13:24:54

Even if I use just node as image name (since it's shorter than ghcr.io/kube-vip/kube-vip:v0.5.5), I still get the same error

Danny Bessems
2023-05-22 14:51:27

Ok, figured it out; somehow packer/ansible/ssh in my build-environment image introduced this new behaviour; I rolled back that image and the issue is no longer present.

I'll try rebuilding with latest versions of all binaries involved and see if the behaviour re-occurs.

Danny Bessems
2023-05-22 15:34:15

Looks like the difference is Ansible 2.14.5 vs 2.15.0; pinning to 2.14.x to avoid the issue for now

👍 jsturtevant
jsturtevant
2023-05-22 18:00:27

can you open an issue about upgrading to ansible 2.15.0?

jsturtevant
2023-05-22 18:00:42

we will need to eventually upgrade.

jsturtevant
2023-05-22 18:00:44

Thanks!

Danny Bessems
2023-05-23 12:25:21

Sure thing, though I'm not using make deps to install stuff, so my environment is not necessarily representative.

👍 jsturtevant
Slackbot
2023-05-22 15:30:01

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma
jsturtevant
2023-05-22 16:17:25

I'll be a few mins late

👍 Marcus Noble
Danny Bessems
2023-05-22 16:01:24

Something I've noticed between building node-templates (.ova's) for vSphere. Ubuntu 20.04 produced 1.7GB large .ova's, Ubuntu 22.04 produces 3.0-3.5GB large .ova's (for reference, RHEL8 is 1.8GB large).

What can explain the increased size between 20.04 and 22.04; should we add a custom role for 22.04 to uninstall a load of new packages (for instance the frustrating needrestart as just a random example)?

mboersma
2023-05-22 20:31:41

PSA: image-builder branch renaming and cruft removal

:megaman: jsturtevant
mboersma
2023-05-22 20:31:52

The image-builder project initially tried to collect VM image-building utilities for Kubernetes in one repository. Over time, the kubedeploy/imagebuilder and konfigadm tools have become unmaintained, while the images/capi area has remained active.

Users and developers are only interested in this latter "Image Builder for Cluster API" area. The presence of the other tools (which have not had code changes in over two years) is an impediment.

We (image-builder maintainers) propose removing these unmaintained projects to simplify the repository. Additionally, we would like to begin using "main" branch nomenclature at roughly the same time as the "cruft removal" described above.

If you have any feedback on these proposed changes, please let us know by commenting on either of the issues listed below before Tuesday, May 30.

Removing unmaintained projects from image-builder · Issue #1143 · kubernetes-sigs/image-builder (github.com)
Rename "master" branch to "main" · Issue #1161 · kubernetes-sigs/image-builder (github.com)

👍 cecile, Julien Klaer
Danny Bessems
2023-05-23 07:53:42

Hurray, then I can stop doing sparse checkouts on that repo 😄

😆 Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2023-05-23 10:46:42

I have just learnt that test-infra has a component called "image-builder" 🙈


This might add some weight to the proposal of also renaming the project to be something like cluster-api-image-builder

GitHub
Danny Bessems
2023-05-23 10:49:12

In that train of thought, openshift has an imagebuilder too

🙈 Marcus Noble, mboersma
Danny Bessems
2023-05-23 10:49:59

It's unavoidable really; descriptive names for building images don't vary a lot 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2023-05-23 10:50:43

True. But if we can be more specific without it being a problem then that might be worth doing 🙂

Danny Bessems
2023-05-23 10:51:25

cluster-api- prefix makes perfect sense to me.

mikejoh
2023-05-23 16:04:55

@mikejoh has joined the channel

Ashutosh Upadhyay
2023-05-25 07:33:00

@Ashutosh Upadhyay has joined the channel

Danny Bessems
2023-05-25 12:47:36

Has anyone got experience with RHEL8 and timing issues with regards to containerd configuration (through - files)?

Danny Bessems (https://kubernetes.slack.com/team/UQU1LL28L)
Shruthi P
2023-05-26 06:54:18

@Shruthi P has joined the channel

Shruthi P
2023-05-26 12:56:54

Hi Team, im trying to create ubuntu capi image. I would like to know how to authenticate it with root/capi user and password?

Drew Hudson-Viles
2023-05-26 12:59:47

You would need to add an SSH public key to it as part of your cloud-init as the password is not generally available. Once you have the key on the machine, you could log in using that and change the password as you wish. Personally I wouldn't recommend that due to the security implications that would introduce.

Shruthi P
2023-05-26 14:38:00

Thanks @Drew Hudson-Viles , pls tell me if this will work - in packer.json there is "ssh-username=builder" . So can i login like ssh builder@IP with password ? to create a image can i use PACKERFLAGS="--var 'kubernetesrpmversion=1.24.0-0' --var 'kubernetessemver=v1.24.0' --var 'kubernetesseries=v1.24' --var 'kubernetesdebversion=1.24.0-00' --var 'disksize=10240'" --var 'sshpassword=enggfusion' --var 'sshusername=enggfusion' make build-kubevirt-qemu-ubuntu-2004

Drew Hudson-Viles
2023-05-26 15:32:02

No problem.

So, the username and password bit you've referred to is for the builder, not the end result of an image - IE these will not be available on the image once you've built it.
See more information on the SSH communicator here.

If you're building from a standard, unaltered Ubuntu image then once the image is built, the only username available by default will be the one supplied by Ubuntu, which is ubuntu

To get your own username and password onto the image, you'd need to provide user-data via the cloud-init "method".

Depending on your infrastructure for creating the VM from this image this can vary.

Danny Bessems
2023-05-30 10:55:55

I've struggled with this exact issue quite a bit (if your nodes do not even get networking configured correctly for instance); my workaround in the end was to temporarily leave the builder account unlocked for debugging, and once issues were resolved replace it again with an image that has the builder account locked.

For reference; change the "shutdown_command" and remove usermod -L {{user ssh_username}} &&

Apricote
2023-05-27 19:21:53

@Apricote has left the channel

Fran
2023-05-31 08:54:17

@Fran has joined the channel

Fran
2023-05-31 09:58:03

Hi folks! I'm trying to get merged, is anyone free to take a look at it please?

GitHub
janr
2023-05-31 10:12:12

@janr has joined the channel

Phani Mvs
2023-06-01 12:18:03

Any idea on who populates this instance-data.json file while provisioning using CAPV provider?
we are seeing issues with RHEL-8 vm templates as ds.meta_data.hostname is getting populated as ‘localhost’

cat instance-data.json 

{
"base64_encoded_keys": [],
"ds": {
"meta_data": {
"hostname": "localhost",
"instance-id": "phani-rhel-26-1-9g5bt",
"local-hostname": "phani-rhel-26-1-9g5bt",
"local-ipv4": "10.109.10.99",
"local-ipv6": "",
"local_hostname": "localhost"
}
}

Jeremi Piotrowski
2023-06-01 12:59:46

cloud-init generates that

👍 Phani Mvs
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-02 09:26:35

I've been spending some time working on the repo cleanup tasks today. I've got the following PRs opened to work towards the cleanup of old projects and the rename of master branch:


❤️ mboersma
:megaman: jsturtevant
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-02 09:27:44

I've also added the full checklist from to the issue to track what is outstanding with the rename -

Danny Bessems
2023-06-02 09:35:05

Will the folder images/capi move to the root of the repo afterwards?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-02 09:41:19

I think we'll leave it for now as not sure what kind of effect that'd then have on all the other projects relying on it.

Danny Bessems
2023-06-02 09:42:01

That makes sense yeah.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-02 09:43:04

Hopefully later we'll be able to move it but I think we should minimize the potential impact right now. 🙂

👍 jsturtevant
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-02 10:27:06

I'm not going to be able to make the office hours on Monday but I've added a couple notes to the existing items on the agenda. Feel free to ping me on Slack with anything needed 🙂

Mohd Waquar
2023-06-05 04:24:14

@Mohd Waquar has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-05 08:10:12

So... turns out that Packer like to introduce breaking changes into patch releases 🤨 We recently bumped Packed from v1.8.6 -> v1.8.7 and that release removed some vendor plugins, specifically for us DigitalOcean. 🤦‍♂️
I opened an issue to cover this:
And a PR to fix it by specifying the external plugin:

The original PR that did the version bump didn't seem to run the pull-packer-validate test that would have caught this. I'll be updating the test in test-infra to make sure this is triggered when changing the ensure script. -

👍 jsturtevant
Danny Bessems
2023-06-05 09:01:21

Hashicorp doesn't follow semantic versioning properly in all of their products 😞

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-05 09:02:42

Gotta keep folks on their toes it seems 😅

Drew Hudson-Viles
2023-06-05 09:09:02

So much just slid into place 🤣

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-05 09:10:42

@Drew Hudson-Viles you should be good to rebase your PR on master now and hopefully should all work 🤞

Drew Hudson-Viles
2023-06-05 09:11:54

Lovely. I'll get that done shortly then. Just fueling myself with coffee after a lovely night of a non sleeping baby.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-05 09:12:15

Ooof! Never fun.

mboersma
2023-06-05 16:17:21

D'oh, and that's why I didn't try to upgrade us to the 1.9.x series, where they mention removing default plugins. Seemed like 1.8.7 was safe, apologies!

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-05 16:18:45

Nah it's ok. Lesson learned 🙂 I thought those tests would have ran on that PR you did so that should have caught it but they weren't actually configured to 🤦‍♂️

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-05 17:49:28

@jsturtevant Mind giving this an /approve ? You should be able to now we've had the owners updated. 🙂

GitHub
Slackbot
2023-06-05 15:30:12

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-05 17:33:51

Is there something up with the test infra? I think this PR is stuck -
Nevermind, sorted now it seems 🙂

feitnomore
2023-06-06 16:43:07

@feitnomore has joined the channel

feitnomore
2023-06-06 16:43:37

Hi... I'm working on building a few cluster images here, and I've got an issue with Rocky Linux 8, which was giving me a 404

feitnomore
2023-06-06 16:43:59

I've submitted a PR fixing the URL, as the URL was looking for 8.7 and the actual version is 8.8

👍 mboersma, Marcus Noble
:thx_thanks: Marcus Noble
feitnomore
2023-06-06 16:44:14

GitHub
feitnomore
2023-06-06 17:33:32

is there any roadmap to add new distros/versions? Like Ubuntu 23.04, Rocky Linux 9?

Christophe Jauffret
2023-06-06 18:54:27

Just open this issue as the commit
is breaking the loadadditionalcomponents feature

GitHub
Drew Hudson-Viles
2023-06-06 22:05:36

So, yes you're right, somehow the defaults are not there which is blowing my mind as I specifically remember putting them in originally. I can only presume I did something in the process of creating the addition that removed them. Probably an erroneous rebase picard_facepalm

I've got fix prepared for this that also adds them "back" in as part of it (though it's not the actual fix).

As for your notes on the aws ansible collection not being officially supported in the current version. It looks like I didn't hardcode the version and an update has occurred since the original PR was created. I'll hardcode the version in to prevent errors as 5.x.x supports 2.11.0+

I'm running a few tests against the fix now it now wrt the issue you've raised which includes me completely removing the collection to ensure it works without it installed, then I'll get a PR put in.

Christophe Jauffret
2023-06-07 07:16:09

perfect don’t hesitate to share me the PR for verification on my side

✅ Drew Hudson-Viles
Drew Hudson-Viles
2023-06-07 08:47:32



I'm just running one last test my side to ensure the Nvidia bits still work (It did last night but it was late and I want a non-tired brain to confirm it) and then we should be good to go.

GitHub
Christophe Jauffret
2023-06-07 08:54:28

PR seems OK, i just launch a test pipeline let’s see

Christophe Jauffret
2023-06-07 10:53:42

ny test pipeline is OK too

Drew Hudson-Viles
2023-06-07 10:54:00

\o/

Christophe Jauffret
2023-06-06 18:56:06

@Drew Hudson-Viles 👆

Drew Hudson-Viles
2023-06-06 18:57:43

Aaah thanks for raising this, I didn't hit this in my local testing but maybe didn't hit the use case of loadadditionalcomponents. I shall take a look into this.

Christophe Jauffret
2023-06-06 18:59:53

there is multiple issues here

  • lack of amazon.aws ansible collection

  • amazon.aws ansible collection is not officially supported with the default image-buidler ansible version 2.11.5

  • new roles values are missing in the images/capi/ansible/roles/loadadditionalcomponents/defaults/main.yml file

Drew Hudson-Viles
2023-06-06 19:06:49

It shouldn't really be affecting you if you're not using the role so I'll see what can be done about that. Ideally you should only need to include the hack file when making use of the S3 role/Nvidia role.

I'll take a look asap to get a fix in.

knfoo
2023-06-07 07:15:30

Hello folks 👋
I have this sitting - any feedback would be great 🙏

knfoo
2023-06-28 14:04:21

Any one ?

Joe Kratzat
2023-06-28 14:27:03

@knfoo I think since the PR is in draft …. it probably won’t get looked at

knfoo
2023-06-28 15:39:13

@Joe Kratzat OK, I was advised to put it into draft first - I can make it a real PR maybe it will get some attention then 🙂

Joe Kratzat
2023-06-28 15:48:33

oh don’t listen to me … I’m not a maintainer … 🤣
Just noticed it was in draft and that typically doesn’t ping people

knfoo
2023-06-28 15:48:58

🙂

mboersma
2023-06-29 21:11:16

Sorry @knfoo we clearly dropped the ball here. I took a look at it and don't see anything controversial.
It would be good to get more eyes on it, but at least one maintainer is on break and July 4 is imminent. Maybe if you promote it from draft status it will notify people?

knfoo
2023-06-30 07:04:02

@mboersma that is OK - I got attention now 🙂
I will promote it from draft and see what happens

👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-08 08:25:27

Do we want to un-pin this issue from the GitHub issues page?

GitHub
👍 mboersma, jsturtevant
jsturtevant
2023-06-08 16:36:41

done! thanks for the reminder

👍 Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-09 09:51:16

I've just noticed we have another OWNERS file here that requests CAPI maintainers to give approvals. See example of confusion on this pr:

Anyone against removing this and just relying on the image-builder OWNERS file. Now we've only got the single project I don't think it makes sense to have both anymore.
/cc @mboersma @jsturtevant

👍 jsturtevant, mboersma, Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-12 11:00:22

PR -

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-12 15:23:37

@mboersma Looks like it needs /lgtm also. I thought /approve implied that but I guess not 😕

Harshil Patel
2023-06-10 16:20:06

@Harshil Patel has joined the channel

Christophe Jauffret
2023-06-12 17:41:44

Hello team, is there a specific release cadency in place for image-builder ? if not what is the rules to trigger a new release ?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-12 17:57:14

It’s pretty ad-hoc right now.
Are you looking for a new release for a specific feature?

Christophe Jauffret
2023-06-12 17:58:41

yes
the EKS image-builder need to consume some change we put in the nutanix packer flavor
and he can only link to a specific tag or a release no more a commit
so we need a new tag or release 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-12 19:17:05

Makes sense.
@mboersma @jsturtevant I think we said recently we wanted to do a release before the branch rename anyway, right?

👍:skin_tone_2: Christophe Jauffret
jsturtevant
2023-06-12 20:45:31

I think that makes sense

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-13 08:27:17

I think one of y'all are going to need to do the release. I'm still waiting on this PR before I have the GitHub perms to create releases 😞

GitHub
Christophe Jauffret
2023-06-13 08:36:24

no perm on my side too 🤷‍♂️

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-13 08:47:26

Sorry, was referring to Matt and James 🙂 They're US based so should be online later. I also need to chase up getting that PR approved so I have the right permissions too.

😅 Christophe Jauffret
mboersma
2023-06-13 18:22:31

I can do a release today, I'll get started in a bit. Thanks for the nudge @Christophe Jauffret.

Christophe Jauffret
2023-06-13 18:27:15

thanks @mboersma

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-13 18:41:49

Need any help @mboersma or you ok to handle it all? 🙂

mboersma
2023-06-13 18:43:10

Thanks, I've got it. I would pair with you @Marcus Noble but I'm in a sig-docs meeting and multitasking. 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-13 18:54:29

No worries 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-13 18:54:34

Thanks for taking care of it 🙂

Ilya Alekseyev (NTNX)
2023-06-12 17:43:40

@Ilya Alekseyev (NTNX) has joined the channel

Sid
2023-06-12 21:38:18

@Sid has joined the channel

mboersma
2023-06-13 19:20:17

Image-builder v0.1.16 is now available:
Thanks to all the (many) contributors! thank_you

🎉 jsturtevant, Marcus Noble, Christophe Jauffret, Mitchel Haring, Julien Klaer, Fran, Drew Hudson-Viles, Ilya Alekseyev (NTNX), cecile
:kubernetes_intensifies: Julien Klaer
hislacker
2023-06-15 13:47:35

@hislacker has joined the channel

hislacker
2023-06-15 13:50:16

Is it possible to use an existing qcow2 image file (which is not created by image-builder), then pass that file as input to image-builder, and image-builder could generate a new image file, which would be cluster API compatible?

jsturtevant
2023-06-15 19:50:37

image builder uses ansible and packer to configure a vm. If there was a packer provisiner for qcow2 maybe? I don't know anything about the format

hislacker
2023-06-15 21:05:47

There's command to make qemu compatible image file via make_qemu command, I can't find a packer provisioner for qcow2.

Christophe Jauffret
2023-06-16 05:27:43

you just need to use an infrastructure provider who can consume qcow2 file as input, i use the Nutanix one like that, of course you need the corresponding infra.
we can also modify the qemu one to work like that i imagine

hislacker
2023-06-16 14:58:36

ok it would be great if there's some existing reference doc for this one..

Christophe Jauffret
2023-06-16 15:19:47

???

you have a Nutanix cluster ?

hislacker
2023-06-16 15:58:05

No. I am not asking if possible to generate an image to be useful on certain platform. I am asking if possible to take an existing qcow2 image, run imagebuilder process on that image to add relevant kubelet/kubeadm/certs in order for that existing qcow2 image to be k8s cluster api compatible. Per above nutanix provider doc, it seems it could generate an image usable for nutanix, but I could not find the place where it could add things to an existing image file.

Christophe Jauffret
2023-06-16 16:20:54

image-builder project generate CAPI ready image for a bunch of platform, there is no generic way
on which platform did you plan to run your image at the end ?

hislacker
2023-06-16 16:26:20

kubevirt

Christophe Jauffret
2023-06-16 18:17:09

ok i would say qemu is the closest target
but it build from an iso installer
what you can do is to modify
packer/qemu/packer.json
and replace all the iso stuff by using the disk_image parameter
look here

once done you can call your corresponding OS target

hislacker
2023-06-16 20:44:13

interesting! Thanks!

🙏:skin_tone_2: Christophe Jauffret
Yike Wang
2023-06-19 09:12:37

Hi, there are multiple case images . Where are they stored? How to understand amifilterowners , is it an account that can reach the base ami? 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-19 09:16:48

I'm not sure I understand what you're asking exactly.
The amifilterowners is used to filter the list of returned AMIs that match the amifiltername. This should result in just the base images we require. All the ones defined in the various .json files we have should be publicly available, if not then we need to fix those.
If you're looking to use your own base image then you can replace amifilterowners and amifiltername with valid filters that will result in your account finding the AMI you want to use.

Is that what you were looking for?

:_thank_you_: Yike Wang
Yike Wang
2023-06-19 09:24:09

I see. Who is responsible to maintain the base amis? And are the base amis avaliable in all the aws regions?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-19 09:34:59

That I'm not sure about unfortunately 😞
Hopefully they are "official" distro images but I'd need to check to be sure.

Yike Wang
2023-06-19 09:36:54

Thanks @Marcus Noble! Looking forward for your checkings 😀

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-19 09:37:40

Out of curiosity, what prompted the question? Are you looking to change something with CAPA?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-19 09:43:27

  • Amazon2 - Official

  • CentOS - Not sure but @swan updated the owner in this PR -

  • Flatcar - Official

  • RHEL8 - Official

  • RockyLinux - Official

  • Ubuntu - Official

  • Windows - Official

👍 richcase, Yike Wang, Paokrab
Yike Wang
2023-06-19 10:16:04

What does "Official" mean? Does it mean they are available across all aws regions or we don't need take care its mantainece or something?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-19 10:17:34

Official here means the account is run by either AWS themselves (AWS Marketplace) or is the account suggested to use by the distro's own documentation.
None of those images are without our control.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-19 10:21:07

**Within our control.

Yike Wang
2023-06-19 10:27:09

Got it, thanks @Marcus Noble!

Yike Wang
2023-06-19 10:29:00

Asking it is because our account that is responsible for making and publishing AMIs is getting migrated. So I think more on the base ami 🙂

👍 Marcus Noble
Slackbot
2023-06-19 15:30:25

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles
2023-06-19 15:32:50

I'll not be around for this one I'm afraid. I've not long got back from my holiday and I'm a bit all over the place atm.

I've not got anything major to raise for this one anyway.

WRT what @Marcus Noble has put in the agenda so far, I'll simply say I agree with all 😄

👍 mboersma, Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-19 18:12:47

@mboersma I took a little look at what we could do with the Makefile include. It's possible but we'd need to do a lot of updating to the Makefile tasks to make use of $(CURDIR) when referencing any files. I'm not confident enough to make all those changes as I don't understand a good chunk of that Makefile 😅
If you're interested, the basic idea would be...
In the root Makefile:

[...existing code...]

export CWD=images/capi
include $(CWD)/Makefile

In the images/capi/Makefile:
CWD ?= ./
CURDIR := $(realpath $(CWD))
(and then update all references to also use $(CURDIR))

mboersma
2023-06-20 16:41:39

@Marcus Noble that does sound a bit fiddly, but not complicated. Maybe we could try it out and see how it acts IRL.
(For context, Marcus and I were discussing flattening the repository, since everything important is nested down in /images/capi. We thought if we could somehow "proxy" the nested Makefile through the root one, it would relieve most of the pain. That is, typing make -C images/capi could be just make and we could avoid the disruption of actually moving those directory contents.)

👍 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-20 16:54:56

I’ll try and get a PR up at some point. The main thing that might cause problems is if we run any scripts that expect the cwd to be images/capi

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-23 08:11:08

@mboersma I finally for to putting the PR together. I'm not 100% sure I've caught everything but the handful of tasks I checked worked as expected.

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-23 08:11:59

As an aside... that Makefile is horrible to work with. There's SO MUCH. 😆

mboersma
2023-06-26 20:50:20

I know, it's huge and hard to navigate. I'll try out the make changes soon, sorry to let it dangle @Marcus Noble. (And thanks!)

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-26 21:20:32

It's ok. There's no rush for it. 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-19 19:03:07

Hey y'all, is anyone still building Ubuntu-18.04 images with image-builder?

We're currently removing it from GCP because the base image is no longer available and wondering if we should remove it from all providers, even those that still have base images available. Ideally we'd like to clear out old OS versions but if there's still a need for it we'll keep it in image-builder for the time being. 🙂

➕ dims
:kubernetes_intensifies: dims
✂️ mboersma
Drew Hudson-Viles
2023-06-19 19:03:49

Not on my side. We're on 22.04 minimum now.

mboersma
2023-06-20 16:27:49

We stopped building 18.04 for CAPZ in April, when it went out of support.
The only reason I haven't made a PR to remove it from azure/ is this:

With an Ubuntu Pro subscription, your Ubuntu 18.04 LTS deployment can receive Expanded Security Maintenance (ESM) until 2028.

mboersma
2023-06-20 16:29:19

So theoretically, there could be an enterprise user with an Ubuntu Pro sub building their own images (as we strongly recommend, the "reference images" we build aren't updated with CVE fixes for one).

mboersma
2023-06-20 16:33:20

Although this theoretical user can and probably has forked the image-builder repo, so they can continue using the 18.04 make targets if we remove them.
I think for the vast majority of use cases, it's at best clutter and at worst a liability to keep all the 18.04-related stuff.
Evidently I've just talked myself into removing 18.04 globally, but I'm curious to hear other opinions still.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-20 16:57:02

I agree with your reasoning. It’s always possible to use an older version of image-builder if needed. Or adding a custom packer json.
If no one comes forward with a strong case for keeping it then let’s clear it out from all providers.

👍 mboersma, Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-19 20:11:03

Could anyone confirm for me if the qemu images can / are ever used on VMWare platforms?
I've opened a pull request to address this issue about open-vm-tools being installed in qemu images due to us sym-linking some files between providers but I want to make sure first that removing this isn't going to cause issues for anyone.

tormath1
2023-06-21 13:51:00

@tormath1 has joined the channel

tormath1
2023-06-21 14:04:21

Hello folks, I'm trying to build a Flatcar image using QEMU builder and it fails with this:

$ make OEM_ID=openstack build-qemu-flatcar
...
qemu:
qemu: PLAY [all] *
qemu:
qemu: TASK [Gathering Facts]

qemu: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: command-line line 0: keyword identitiesonly extra arguments at end of line", "unreachable": true}
qemu:
qemu: PLAY RECAP
*
qemu: default : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0
qemu:
==> qemu: Provisioning step had errors: Running the cleanup provisioner, if present...
==> qemu: Deleting output directory...
Build 'qemu' errored after 7 minutes 55 seconds: Error executing Ansible: Non-zero exit status: exit status 4
It was working fine previously, I suspect the OpenSSH upgrade of my system:
$ ansible --version
ansible [core 2.15.1]
config file = /home/mathieu/github/kubernetes-sigs/image-builder/images/capi/ansible.cfg
configured module search path = ['/home/mathieu/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/mathieu/github/kubernetes-sigs/image-builder/images/capi/.env/lib/python3.11/site-packages/ansible
ansible collection location = /home/mathieu/.ansible/collections:/usr/share/ansible/collections
executable location = /home/mathieu/github/kubernetes-sigs/image-builder/images/capi/.env/bin/ansible
python version = 3.11.3 (main, Jun 2 2023, 13:54:39) [GCC 12.2.1 20230428] (/home/mathieu/github/kubernetes-sigs/image-builder/images/capi/.env/bin/python)
jinja version = 3.1.2
libyaml = True
$ ssh -V
OpenSSH_9.3p1, OpenSSL 1.1.1u 30 May 2023

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-21 14:24:10

Are you using the latest image-builder from the master branch or a tagged release?
Are you able to check if you experience the same issue using the Docker image?

tormath1
2023-06-21 14:47:25

I'm on master
Let me check with Docker but I need to rebuild an image to use root user instead of imagebuilder (see: )

tormath1
2023-06-21 15:26:46

That seems better from Docker - I see the image is using a different OpenSSH version:

OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022
Let me see if I can get the faulty command.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-21 15:28:15

Ok, so that at least confirms that the issue is related to OpenSSH being updated. Not sure how we'll tackle that thinking

tormath1
2023-06-21 16:01:30

Mm, maybe Ansible actually. I found this:

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-21 16:03:50

Hmm... so maybe we just need to bump Ansible to v2.15.1?
Do you mind trying again with this value bumped up?

GitHub
tormath1
2023-06-21 16:07:56

I'd say the opposite actually: the version 2.15.1 does not currently work with image-builder.
Let me try with your version !

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-21 16:11:10

Oh sorry, I misunderstood.

tormath1
2023-06-21 16:22:23

That works fine with 2.11.5 so it's not OpenSSH related but Ansible related

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-21 16:24:32

Ok. That's good to know at least. Hopefully it gets resolved in Ansible before you do another upgrade of it in image-builder.

tormath1
2023-06-21 16:29:10

I'm trying to see if we could not send a similar patch: for SSH extra arguments ()

tormath1
2023-06-21 16:40:31

Anyway, thanks for your help 💪

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-21 16:43:47

No worries 🙂 Glad you managed to figure it out. 😄

Wolodja Wentland
2023-06-21 16:49:14

@Wolodja Wentland has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-22 12:56:28

📣 Announcement!
I am about to begin the master -> main branch rename as outline in this issue.
If all goes smoothly nothing should be effected as PRs and git references should update automatically. If you notice any problems please let me know in the thread. 🙂
I'll announce again once complete so people can update their own checkout out repos / forks if they choose.

:emoji_k8s_loft: Drew Hudson-Viles, Ricky Sadowski, mboersma, jsturtevant
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-22 12:57:13
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-22 12:58:19

To update your local checkout out copy:

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-22 13:05:55

😒 The test-infra PR deciding to re-run all the checks again after removing the hold label is not ideal.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-22 13:22:48

Held PRs now merged.
Just waiting on confirming that CI jobs still work as expected. (See )

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-22 13:38:58

Both PR and periodic jobs are able to run correctly! 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-22 13:40:11

Rename complete! 🎉
If you experience any issues you believe is related to the rename please don't hesitate to let us know. 🙂 If you'd like to update your local copy there's commands in this thread you can run.

:thank_you: mboersma, jsturtevant
:high5: Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-22 13:41:14

I've opened a PR in test-infra to allow image-builder maintainers to re-run our periodic jobs via Prow -

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-22 13:56:06

@mboersma Did you get anywhere with the Netlify config update? The book is still working (as I expected) but I'm not sure about new changes being built and deployed. I've opened a cleanup PR that we can use to test the change when we're ready to do so.

GitHub
mboersma
2023-06-22 17:49:32

I think it's all done. We can merge the cleanup PR to make sure.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-22 18:35:41

Already done 🙂 And confirmed it worked! 😄

👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-22 18:36:11

Oh, just realised you sent this before I did the PR 😆 My notifications don't seem to be working right.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-26 08:26:21

Hey y'all, I'd like to get peoples thoughts on using rolling-version base images (within a set major version) and if people think this is appropriate or not for image-builder?
The question comes from this PR and originally this comment. The RockyLinux links we're currently using in Nunatix are dead but have been moved elsewhere. The PR is suggesting changing them to point to the latest of the major release which would mean we don't need to keep updating the value but it does mean that re-runs of the same image-builder configuration could result in different images being built. What does everyone think?

Christophe Jauffret
2023-06-26 08:29:17

Actually Image-builder inside his OS task is performing yum update / apt update during each build.
so each result is different with always the last security update.
Using the rolling-version as source will give the exact final result and even better will improve the build time because no need to download package twice.

so i approved :-D

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-26 08:32:27

Yeah, that is actually a really good point. I'm not sure we actually do this with other OSs (e.g. Ubuntu) but we definitely do with RHEL.

Christophe Jauffret
2023-06-26 08:33:29

same with ubuntu

- name: perform a dist-upgrade
apt:
force_apt_get: True
update_cache: True
upgrade: dist
register: apt_lock_status
until: apt_lock_status is not failed
retries: 5
delay: 10

Christophe Jauffret
2023-06-26 08:33:45

in images/capi/ansible/roles/setup/tasks/debian.yml

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-26 08:33:49

Yup! Just found that as you posted 🙈

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-26 08:33:59

Ok, I'm being convinced 😆

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-26 08:34:30

Lets leave it a little just in case anyone wants to weigh in but if not I'm happy to approve that PR 🙂

Christophe Jauffret
2023-06-26 08:35:40

security first, and also sustainable IT improvement 😎
…. and i don’t like wait

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-26 08:45:36

The other downside I see to using the lastest base image is we can't use checksum to ensure the image is as expected. But was we're not currently doing that for Nutanix anyway I guess it doesn't matter.

Christophe Jauffret
2023-06-26 08:46:46

yes agree

Christophe Jauffret
2023-06-26 08:47:47

was thinking to implement external checksum file support for this kind of case

mboersma
2023-06-26 16:26:43

I'm ok with using latest, because we already don't have 100% recreatable builds, since Ubuntu and others can already update underneath us (and I was planning to use "latest" for Mariner Linux support).

The tradeoff is obviously that a given CI run might break, but we already see that occasionally (usually from an old distro rusting too much, like Rocky Linux here).

👍 Marcus Noble, jsturtevant
👍:skin_tone_2: Christophe Jauffret
Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-26 16:57:18

Works for me 🙂 I'm convinced. Thanks!

Marcus Noble (k8s@marcusnoble.co.uk)
2023-06-26 16:58:20

lgtm approval

👍 mboersma
Ivo Perruci Neto
2023-06-26 18:06:21

@Ivo Perruci Neto has joined the channel

Ozhan Karaman
2023-06-26 20:39:45

@Ozhan Karaman has joined the channel

Slackbot
2023-07-03 15:30:08

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2023-07-03 16:35:09

Skipping the office hours this week due to low attendance and nothing pressing to discuss. 🙂

Jan
2023-07-05 19:47:56

@Jan has joined the channel

Krish Jain
2023-07-06 07:56:51

@Krish Jain has joined the channel

Patrick Enoux
2023-07-08 12:51:36

@Patrick Enoux has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2023-07-09 14:31:10

Could I please get a review on this small fix for Mac M1/2 users? 🙏
https://github.com/kubernetes-sigs/image-builder/pull/1215

GitHub
:lgtm: mboersma
mboersma
2023-07-10 16:09:15

@Marcus Noble thanks for that!
Maybe we should do an image-builder release soon? Several good fixes, plus Mariner Linux and maybe updating to latest packer, seems like a good milestone to release.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-07-10 16:10:16

👍 Sounds good to me.
Anything else in the open PRs that'd be good to get in?

mboersma
2023-07-10 17:42:32

Nothing that jumps out at me. Maybe we wait for to merge and then do a release.

👍 Marcus Noble
mboersma
2023-07-12 21:15:42

I think we're at a good point to do a v0.1.17 image-builder release. I can kick that off this afternoon or tomorrow if we have consensus.

mboersma
2023-07-13 03:03:59

I tagged v0.1.17, here's the image promotion PR:

:thx_thanks: Marcus Noble
:approval: Marcus Noble
rahav jv
2023-07-12 14:15:04

Folks, I'm trying to set cdromtype to sata as a packer variable. I have set this is in a file which is passed to PACKERVAR_FILES . However when the packer vm comes up, the vm comes up with cdrom set as ide. Any suggestions on what i might be missing

kiran keshavamurthy
2023-07-12 19:03:32

Hey Rahav, cdromtype is not used in any of the builders so I guess it just defaults to ide . Hence passing the type has no effect. I suggest adding cdromtype var to the builders in packer-node.json , and pass the type from the from the os configs. For Photon-5 you might need sata I guess.

Debjit
2023-07-12 15:35:05

@Debjit has joined the channel

Debjit
2023-07-12 15:45:05

I could not understand the usage of image-builder Can't we use any ami with cluster-api ?

Drew Hudson-Viles
2023-07-12 15:49:00

image-builder is designed to set a bunch of sensible defaults as well as install the tools required for it to work with capi.
On top of that it supports multiple clouds, multiple distros and can be configured to install additional container images, nvidia drivers and more.

A base ami, or any other image from any other provider would require all of the bits the image builder project adds to be done manually.

It just takes the toil out of building images really.

Debjit
2023-07-12 15:51:24

Thanks @Drew Hudson-Viles. So we just cannot use any ami with cluster-api , right?

Drew Hudson-Viles
2023-07-12 15:51:56

Well you could, but you'd need to install things on top of it, like kubelets, containerd (or alternative) etc.

Drew Hudson-Viles
2023-07-12 15:52:19

capi isn't something you can just install and get all those things with it 🙂

Debjit
2023-07-12 16:58:40

@Drew Hudson-Viles Ok. Which means Cluster API does not install kubeadm and necessary applications and tools by itself. Like in the case of Kubespray.

Drew Hudson-Viles
2023-07-12 17:11:09

Yeah that's right. The best way to think about it is that Cluster API is just a collection of APIs that allow you to manage multiple clusters from a single management cluster - that could be a full blown one or kind that you run locally.

Debjit
2023-07-13 09:38:50

@Drew Hudson-Viles Got it. Thanks.

Debjit
2023-07-13 23:23:51

@Drew Hudson-Viles We can only use ami build for cluster-api Reading through the image-builder book and it seems it is not an easy route to build custom ami. As an example, i need to build aarch64 based ami as we are using AWS Graviton processor based instances, and there seems to be no way to build the required ami with image-builder at the moment. Can you advise some alternatives. Any documentation what needs to be provisioned for ami to be compatible with cluster-api , then I can probably create an ami manually for the time being.

Drew Hudson-Viles
2023-07-14 09:14:24

I can't say for certain as I've not used graviton to date and I'm out of AWS at the moment so can't do any testing.

You may be able to modify this to get it to work with graviton based AMIs but without any testing I cannot confirm for sure. Depending on the package manager it uses etc it may be required you use a different one.

If you do want to do it manually for now then there is a decent amount of documentation available. I'd recommend taking a look through the CAPI book and the cluster-api-provider repo

In the AWS CAPI provider readme there is information about using pre-baked AMIs which I'd suspect Amazon already provide to save any leg work on your side.

The final option I'd say is to read through the Ansible playbooks in the image builder repo and see what's being done at each stage and attempt to replicate that as part of the graviton AMI.

Debjit
2023-07-14 12:37:59

@Drew Hudson-Viles Your thoughts align. Thanks. The only issue is in image-builder the amd64/x86_64 is hardcoded. Let me do a little brainstorming and will let you know the results.

👍 Drew Hudson-Viles
mboersma
2023-07-13 15:37:17

Image-builder v0.1.17 is now available:
Thanks to all the contributors! thank_you

:parrotk8s: Marcus Noble, jsturtevant
:thx_thanks: Marcus Noble
🎉 Marcus Noble, Ozhan Karaman
Debjit
2023-07-14 12:39:22

How can I build arm64/aarch64 ami with image-builder?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-07-14 12:56:03

There's actually a PR I've just approved to add that support:

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2023-07-14 12:57:25

Once that's merged in you should be able to by setting the kubernetes_goarch var. I think you also need to change the base image you use too. I'm not sure as I haven't done it myself.

Debjit
2023-07-14 23:39:39

👍

Debjit
2023-07-16 22:42:33

@Marcus Noble I cloned the repo but could not find kubernetes_goarch var

Marcus Noble (k8s@marcusnoble.co.uk)
2023-07-16 23:57:17

If you add kubernetes_goarch to your user provided packer vars then it’ll use the value you provide rather than the default. (See here for details on how if you’re not sure: https://image-builder.sigs.k8s.io/capi/capi.html#customization)

Debjit
2023-07-18 16:56:44

@Marcus Noble What about other hard coded values? There are a lot of places where am64/x86_64 is hard coded.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-07-18 17:33:01

Sorry, I didn’t realise there was still more outstanding work. Looks like we have this issue open to track it: https://github.com/kubernetes-sigs/image-builder/issues/936

GitHub
Tham Xun Hong
2023-07-14 15:48:25

@Tham Xun Hong has joined the channel

Debjit
2023-07-16 22:41:48

Can we have prebuilt image for Amazon Linux 2023 with Kubernetes v1.27.3. Tried building one with image-builder but epel package install failed, since AL 2023 does not allow the same.

Debjit
2023-07-18 17:15:43

@Marcus Noble Sure

Marcus Noble (k8s@marcusnoble.co.uk)
2023-07-17 07:59:03

Is anyone able to take a look at this PR and share your thoughts please? 🙂

GitHub
👍 mboersma
Slackbot
2023-07-17 15:30:20

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma
2023-07-17 17:10:17

Thanks for attending everyone! If you couldn't attend, a recording of the office hours is available on YouTube here:

Debjit
2023-07-18 17:15:00

@Marcus Noble How can I make the images private only?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-07-18 17:30:03

What do you mean? What provider are you building for? What vars etc?

Debjit
2023-08-08 22:33:29

@Marcus Noble AWS. It is solved now.

Igar V
2023-07-19 15:59:54

@Igar V has joined the channel

Erkan Erol
2023-07-20 11:33:53

@Erkan Erol has joined the channel

Erkan Erol
2023-07-20 11:40:58

Hello. In the OVF file of Flatcar image for CAPV, variables for userconfig are missing. Is it intentional? While testing images via UI, they are very useful.

Here is the related place. I expect to see these variables there.

Erkan Erol
2023-07-20 11:43:04

Here is the related part in flatcarproductionvmware_ova.ovf





Flatcar Container Linux Virtual Appliance
ovf:key="guestinfo.hostname" ovf:value="">

Hostname

ovf:key="guestinfo.ignition.config.data" ovf:value="">

Inline Ignition config or coreos-cloudinit data (cloud-config or script)

ovf:key="guestinfo.ignition.config.data.encoding" ovf:value="">

Encoding for Ignition config or coreos-cloudinit data (e.g., base64)

ovf:key="guestinfo.ignition.config.url" ovf:value="">

URL to Ignition config or coreos-cloudinit data (cloud-config or script)

ovf:key="guestinfo.dns.server.0" ovf:value="">

Primary DNS (only for coreos-cloudinit)

ovf:key="guestinfo.dns.server.1" ovf:value="">

Secondary DNS (only for coreos-cloudinit)

ovf:key="guestinfo.interface.0.name" ovf:value="">

Name for network interface 0 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.0.mac" ovf:value="">

MAC for network interface 0 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.0.dhcp" ovf:value="no">

DHCP support for network interface 0 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.0.role" ovf:value="public">

Role for network interface 0 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.0.ip.0.address" ovf:value="">

Main IP for network interface 0 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.0.ip.1.address" ovf:value="">

Additional IP for network interface 0 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.0.route.0.gateway" ovf:value="">

Main route gateway for network interface 0 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.0.route.0.destination" ovf:value="">

Main route destination for network interface 0 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.0.route.1.gateway" ovf:value="">

Additional route gateway for network interface 0 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.0.route.1.destination" ovf:value="">

Additional route destination for network interface 0 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.1.name" ovf:value="">

Name for network interface 1 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.1.mac" ovf:value="">

MAC for network interface 1 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.1.dhcp" ovf:value="no">

DHCP support for network interface 1 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.1.role" ovf:value="private">

Role for network interface 1 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.1.ip.0.address" ovf:value="">

Main IP for network interface 1 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.1.route.0.gateway" ovf:value="">

Main route gateway for network interface 1 (only for coreos-cloudinit)

ovf:key="guestinfo.interface.1.route.0.destination" ovf:value="">

Main route destination for network interface 1 (only for coreos-cloudinit)



nickperry
2023-07-21 00:26:03

@nickperry has joined the channel

nickperry
2023-07-21 00:28:35

In Kube 1.27, the in-tree kubelet credential provider for AWS was removed (). This followed GA of the external kubelet credential provider feature in 1.26.

At my organisation we pull most of our images from private ECR repos, so since this removal in 1.27, we need the external ecr-credential-provider binary in our CAPV OVAs.

What are the maintainers thoughts on including external credential provider binaries (such as ecr-credential-provider) in published images, such as the OVAs distributed at ?

nickperry
2023-07-21 08:40:06

Thanks. The ecr-credential-provider binary is 21MB.

nickperry
2023-07-21 08:41:08

I need to double check, but I don't think CAPA have included it yet either.

xd
2023-07-21 10:29:20

@xd has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2023-07-24 06:54:24

I've opened an issue to track removing old end of life OSs from image-builder defaults:

GitHub
👍 mboersma
Dharmjit
2023-07-26 15:31:08

Hi Folks, I was trying build-node-ova-vsphere-photon-5 target to build ova with a different photon iso but the script is stuck at step vsphere-iso.vsphere: Waiting for SSH to become available... . When tried with the higher verbosity using FOREGROUND=1 PACKERLOG=1 , I get below in the output. I am running scripts in Ubuntu 20.04.5 LTS machine and using vSphere7 as the hypervisor.
Not sure how packer works but I could see the ssh
username and ssh_password are already defined in images/capi/packaer/ova/packer-common.json. Any inputs?

==> vsphere-iso.vsphere: Waiting for SSH to become available...
2023/07/26 14:16:13 packer-builder-vsphere-iso plugin: [INFO] Attempting SSH connection to 10.xx.xx.xx:22...
2023/07/26 14:16:13 packer-builder-vsphere-iso plugin: [DEBUG] reconnecting to TCP connection for SSH
2023/07/26 14:16:13 packer-builder-vsphere-iso plugin: [DEBUG] handshaking with SSH
2023/07/26 14:16:17 packer-builder-vsphere-iso plugin: Keyboard interactive challenge:
2023/07/26 14:16:17 packer-builder-vsphere-iso plugin: -- User:
2023/07/26 14:16:17 packer-builder-vsphere-iso plugin: -- Instructions:
2023/07/26 14:16:17 packer-builder-vsphere-iso plugin: -- Question 1: Password:
2023/07/26 14:16:19 packer-builder-vsphere-iso plugin: [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password keyboard-interactive], no supported methods remain
2023/07/26 14:16:19 packer-builder-vsphere-iso plugin: [DEBUG] Detected authentication error. Increasing handshake attempts.
2023/07/26 14:16:26 packer-builder-vsphere-iso plugin: [INFO] Attempting SSH connection to 10.xx.xx.xx:22...
2023/07/26 14:16:26 packer-builder-vsphere-iso plugin: [DEBUG] reconnecting to TCP connection for SSH
2023/07/26 14:16:26 packer-builder-vsphere-iso plugin: [DEBUG] handshaking with SSH

Dharmjit
2023-07-26 17:32:35

@kiran keshavamurthy tagging you if you can provide some inputs.

shravanr
2023-07-26 16:30:15

@shravanr has joined the channel

Slackbot
2023-07-31 15:30:17

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma
Max Fedotov
2023-08-01 10:03:17

@Max Fedotov has joined the channel

opatrick
2023-08-07 11:54:45

In some Docker Images i found an spdx file, what is this file for?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-07 12:51:55

What file specifically are you referring to? I don’t know of any spdx files but you can find the Dockerfile here: https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/Dockerfile it might give you some hints.

Jeremi Piotrowski
2023-08-07 14:08:54

spdx is a format for SBOMs (software bill of materials) so it would be metadata about contents and licenses of package inside a container image

❤️ opatrick
opatrick
2023-08-07 14:14:05

Ahh cool, is this a docker feature?

Jeremi Piotrowski
2023-08-07 15:49:31

im not sure what the source of the sbom is that you're seeing

Jeremi Piotrowski
2023-08-07 15:49:36

docker has some form of integration

Jeremi Piotrowski
2023-08-07 15:49:41

Docker Documentation
❤️ opatrick
Abhay Krishna Arunachalam
2023-08-10 23:41:04

Does this raise any concerns for the usage of Packer in image-builder?
https://discuss.hashicorp.com/t/hashicorp-projects-changing-license-to-business-source-license-v1-1/57106
https://www.hashicorp.com/blog/hashicorp-adopts-business-source-license

HashiCorp Discuss
HashiCorp
Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-11 08:30:40

Somewhat, yes.
I don’t understand enough to know if it will cause problems for image-builder but I suspect it could do.
I’ve added it as an agenda item to the next office hours meeting on Monday.

👍 jsturtevant, mboersma
jsturtevant
2023-08-11 16:45:30

we may need to get some help from the CNCF folks

mboersma
2023-08-11 17:09:50

Related: Bump Packer to v1.9.2

👍 Marcus Noble
:thx_thanks: Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-11 17:20:46

https://kubernetes.slack.com/archives/C5P3FE08M/p1691750465350369?thread_ts=1691699636.105219&channel=C5P3FE08M&message_ts=1691750465.350369

Davanum Srinivas (https://kubernetes.slack.com/team/U0Y7A2MME)
👍 jsturtevant
Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-11 17:21:26

I’m AFK until Monday. If someone else has the time before then would you mind opening an issue to track this?

mboersma
2023-08-11 17:23:41

Sure, I can open an issue. Thanks Marcus!

:thx_thanks: Marcus Noble, jsturtevant
Abhay Krishna Arunachalam
2023-08-11 23:17:23

Thank you @Marcus Noble and @mboersma! I shall keep an eye on the issue

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-15 15:20:06

CNCF have published some guidance:

Doesn't really help us though. 😞 The two recommendations are:

  • Switch to an alternative - Is there one for Packer?

  • Freeze the component version - We've done this but it's a short-term solution at best

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-15 15:24:38

See also:

Abhay Krishna Arunachalam
2023-08-15 17:22:39

If the restriction is on using Packer directly, could one atleast import packer plugins and SDKs as a Go dependency to build one's own CLI or is that out of bounds too?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-15 17:27:44

I haven’t looked. I’m not sure what, if any, libraries there are and I’d they provide enough to replace what we currently do. It’s also possible the libraries are under the new license too as not all has remained under the old license.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-15 17:32:25

Just took a quick look, as far as I see ask the Packer code is now the BUSL license. 😔

😢 Abhay Krishna Arunachalam
:sad: mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-17 09:01:23

Opened license exception request:

GitHub
🙏 Abhay Krishna Arunachalam
🤞 Abhay Krishna Arunachalam
Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 09:48:34

@Drew Hudson-Viles Do you have an example of how one might make use of loadadditionalcomponents? I'm trying to see if I can make use of it in our current builder pipelines at Giant Swarm without too much changes but I'm not sure 😬

Drew Hudson-Viles
2023-08-14 09:49:25

I believe I do, one tick I'll have a look though my various testing I did.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 09:50:16

Thanks 🙂

Drew Hudson-Viles
2023-08-14 09:51:01

Just need to flick my PC on as they are all on there. give me 2-3 minutes

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 09:51:14

No rush 🙂

Drew Hudson-Viles
2023-08-14 09:58:57

So you have to enable the 'additional component' as well as the the role itself with a couple of params.
For example, I have this line to add additional container images into an image - I've omitted the many, many additional ones to make it more readable 😄

"ansible_user_vars": "load_additional_components=true additional_registry_images=true additional_registry_images_list=docker.io/k8scloudprovider/openstack-cloud-controller-manager:v1.25.0,docker.io/k8scloudprovider/cinder-csi-plugin:v1.25.0,k8s.gcr.io/sig-storage/csi-attacher:v3.4.0, install_falco=true install_trivy=true"

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 10:01:13

Nice! 😄 That's actually much easier than I thought. We'd want to make use of additionalexecutableslist and additionalexecutablesdestinationpath then 🙂

Edit: Oh and additional
executables=true

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 10:01:59

🤔 It'd be nice if we could specify different destinations for different executables but as we currently only need 1 we can work with that I think

Drew Hudson-Viles
2023-08-14 10:03:44

Yeah that's exactly it.
I was thinking that too when I played with the additional executables but we'd have to consider how that'd work in terms of providing parameters. It'd be dirty but we could do a destination list where each list item uses the same index as the executable list.... but it's dirty and prone to error 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 10:05:53

Could format it similar to the volume arg with docker - e.g. ${downloadurl}:${targetpath},....

Drew Hudson-Viles
2023-08-14 10:06:22

Yeah that's true

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 10:07:02

But that can wait.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 10:07:33

It'd also be nice to be able to do things like checksum validation etc. but that gets complicated very quickly 😆

Drew Hudson-Viles
2023-08-14 10:07:52

hahaha indeed it does!

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 10:12:17

oh! 🤦‍♂️

We need to also unpack the downloaded tar. UGH!

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 10:14:27

How do other people go about installing additional agents / executables on their images created with image-builder?

Drew Hudson-Viles
2023-08-14 10:15:33

aaah yeah I hit that problem... We do need to think about how we support that process but again it's a complex one due to all tars being packaged differently 😞

I don't have a solution to that one right now.

Danny Bessems
2023-08-14 14:27:35

In my ansible playbook where I encounter that same challenge for a list of binaries I'm injecting, I use an optional parameter like: extra_opts: --strip-components=2 (so far I've actually only used it for --strip-components :P)

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 14:35:42

You're not doing that with image-builder currently though right? As far as I know there's no way of passing in extra archives to extract anywhere? You're referring to the unarchive action not creating a subdirectory, yeah?

Danny Bessems
2023-08-14 15:10:50

Correct, that's not in image-builder's task. But in one of my own ansible playbooks:

  ansible.builtin.unarchive:
src: "{{ item.url }}"
dest: "{{ archive.path }}"
remote_src: yes
extra_opts: "{{ item.extra_opts | default(omit) }}"
But since these additional_components vars are just a list of strings passed through packer to ansible, it gets more complicated where we'd have to make it accept complex patterns

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 15:11:40

👍 Thought so, just wanted to double check I wasn't missing something 🙂 Thank you

Danny Bessems
2023-08-14 15:13:04

I guess :: pattern would make some sense since you can make them optional parts

Danny Bessems
2023-08-14 15:13:16

Worth an issue and a PR 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 15:14:21

Yeah. I think it needs designing first in an issue and agree on what would be needed. This is needed by another team at my company so it might be I get them to contribute it to image-builder if they don't find another solution. 🙂

Danny Bessems
2023-08-14 15:16:32

You could also just add a custom-role in image-builder and have that do the download/extraction/installation

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 15:19:14

We currently make use of the container image of image-builder in a Tekton pipeline to build our images. Adding in a custom role is just as complicated for us currently 😆

Danny Bessems
2023-08-14 15:19:22
  "node_custom_roles_post": "custom",
"node_custom_roles_pre": "",
Danny Bessems
2023-08-14 15:20:03

You'd have to mount the custom roles in your container and then reference it; that's about it?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 15:21:29

🤔 hmm.... actually, that might be straight forward. We could just store the roles in a repo somewhere and have them checked out to a shared workspace in our pipeline.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 15:21:41

/cc @Puru Tuladhar ☝️

👍 Puru Tuladhar
Danny Bessems
2023-08-14 15:22:34

If you go that route, share what you did. Always curious to see how people solve these kind of customizations 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 15:25:21

I still think there might be a use case for including the ability in image-builder directly but would need to think about how the interface for it looks as it could be really messy. I know people have asked in the past about installing security related agents into images.

Danny Bessems
2023-08-14 15:26:52

Wouldn't most of those kinds of agents need config anyway, and then it would already deserve a custom role to cover the whole thing?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 15:27:29

Yeah that's what I'm thinking, it gets too messy and I'm not sure if there's an easy way to handle that kind of thing.

Danny Bessems
2023-08-14 15:29:05

Perhaps we should just document the 'workaround'; though you can't add comments in Packer's json files (reason to migrate to hcl 😇)

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 15:31:19

reason to migrate to hcl
There's an issue for that:

And yeah, the docs could do with more examples and content in general to be honest. Once we get something I'll be sure to get it added into the docs somewhere.

GitHub
👍 Drew Hudson-Viles
Puru Tuladhar
2023-08-14 09:57:15

@Puru Tuladhar has joined the channel

👋 Marcus Noble, mboersma
Slackbot
2023-08-14 15:30:03

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-14 15:31:51

I'm going to be (hopefully) joining from my phone as I'll be out but I might be a few minutes late.

👍 mboersma
Zach Wachtel
2023-08-14 18:50:06

@Zach Wachtel has joined the channel

👋 mboersma, richcase, jsturtevant
mboersma
2023-08-15 14:42:56

announce Image-builder v0.1.18 is now available:
Thanks to all contributors!

🙌 Drew Hudson-Viles, jsturtevant, willie
Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-15 15:58:09

I am liking that we're seeing a good number of new contributors with each release! 😄 💙

👍 mboersma, jsturtevant
salisbury_joe
2023-08-15 16:16:59

@salisbury_joe has joined the channel

👋 Marcus Noble
:giantswarm: Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-17 11:14:57

📣 PR tests are currently failing due to this:
Please bare with us while we get this fixed. Until then you can assume that failures do to Nutanix are ok. Once we have the fix in place PR tests can be re-run.

GitHub
Christophe Jauffret
2023-08-17 11:20:33

Hi @Marcus Noble i let some comment on the issue, open to discuss if needed

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-17 11:21:35

If you're able to handle the requirements needed for v0.8.0 that'd be awesome. I don't know anything about Nutanix so went for pinning the version to what was previously passing for now.

Christophe Jauffret
2023-08-17 11:22:43

yes i can do it
merge your pinning if you want and i will overwrite it in the next PR with the fix tommorow

❤️ mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-17 11:23:02

Perfect! 😄 Thank you!

👍:skin_tone_2: Christophe Jauffret
Dimple Raja Vamsi Kosaraju
2023-08-23 13:30:32

@Dimple Raja Vamsi Kosaraju has joined the channel

naadir
2023-08-23 13:31:32

Hi folks, would anyone be willing to co-sponsor @Dimple Raja Vamsi Kosaraju’s membership of k8s-sigs, who has been contributing a fair amount to image builder of late?

:thankyou: Dimple Raja Vamsi Kosaraju
jsturtevant
2023-08-23 19:19:39

sure megaman it looks like you've been active over a long time period ()

:thanks: naadir
Dimple Raja Vamsi Kosaraju
2023-08-24 06:17:18

thank-you-very-much @naadir and @jsturtevant for sponsoring, will go ahead and raise a Issue for joining the k8s-sigs org

Dimple Raja Vamsi Kosaraju
2023-08-24 06:24:11

Raised Issue#4408 🙂

GitHub
Slackbot
2023-08-28 15:30:18

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles
2023-08-28 15:32:39

I won't be able to make today's unfortunately as I'm out with the family for the bank holiday.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-28 15:47:41

I have a few topics but I suspect that today might be quite as I know Matt is also out today. None of my topics are urgent, mostly just updates, so I’ll postpone if no one else ends up joining.

👍 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-28 17:28:46

The meeting notes (above) have been updated with what was discussed today.

Main discussion points:

  • HashiCorp changes - work still ongoing but we're fairly sure that our use of Packer is valid with the new license (though please be sure to check with your own legal team if possible). The main outstanding piece right now is getting an exemption from CNCF to use a dependency with the BSL license. See

  • PR tests flakes - We're aware and trying to get them resolved! The issue is related to GitHub rate limiting due to the new test cluster having less public IPs than the old. For now please be patient and re-run the tests that fail due to rate limiting. We have an issue to track it and there is discussions going on to get a GitHub token added to the cluster that projects could make use of.

  • Dependency management - I raised the suggestion of introducing automated dependency version PRs. I've wrote up an issue with some of the details here: Please contribute your thoughts as I think it would be great if we could get image-builder following the latest releases of all our dependencies automatically rather than when people need a newly introduced feature.

  • Ansible & Python - We've recently bumped the version of Ansible that we use which now requires Python 3.9. While this is handled in the container image we provide, those running the pipelines directly might experience an issue (see ). For now, we recommend people use the latest tagged release of image-builder or the container image if they're not able to upgrade Python in their environment. Before we make the next release we'll be introducing some handling to ensure the required version of Python is found on the system, similar to how we handle other dependencies.


Once I have access to the office hours recording I'll add it to the meeting notes and update here. 🙂

👍 mboersma, jsturtevant, richcase
arvind
2023-08-28 16:39:20

@arvind has joined the channel

Dipesh
2023-08-29 00:23:50

@Dipesh has joined the channel

Abhay Krishna Arunachalam
2023-08-30 18:47:33

Hello folks, is there a way to run image-builder in an airgapped environment? AFAIK Internet access is required for:

  1. Github - ensure scripts, pulling containerd and other binaries from releases

  2. OS packages - debs, RPMs

  3. Kubernetes container images

and many other things.
So how does one do image builds from a machine with no internet access?

Drew Hudson-Viles
2023-08-30 18:58:27

Hi! 👋

I think you're going to struggle to do this with the repo to be honest. I think if you really wanted to do this you'd have to fork the repo into your environment and make some pretty significant changes to it to get it to use your own package repo, images etc. But then you risk it falling out of line with everything else.

I'm not sure if anyone else has experience with this and can help further, but as the internet is required in its current form I'm not sure how to recommend proceeding with this.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-08-30 19:15:45

As Drew said, the answer is pretty much “no”. There’s too much external dependencies that would need to be replicated locally it wouldn’t be worth it. It would be less work to build a custom script to handle your specific use case.

I’m curious as to way building of the images must be done in an air gapped environment. Could you explain more your use case?

Abhay Krishna Arunachalam
2023-08-30 22:06:32

Thanks for your insights Drew and Marcus! We are trying to get Image-builder working offline with Artifactory repos for debs/RPMs, local registry for container images, local mirrors for binaries/executables pre-downloaded from github. Wanted to know if that's even a feasible path

Danny Bessems
2023-09-01 10:27:44

Well of course you technically can use your own mirrors for everything, it'll just be a lot of work to keep your changes working with every new image-builder release I think.

I'd start looking at creating a custom role and referencing it in as a pre ansible role. In there you'd have to change configuration of all relevant repositories.
But that won't cover other tasks in the existing roles that reference their own sources, so you're going to have to go through all these other roles to see if you can override sources that are in use in a reproducible way.

kiran keshavamurthy
2023-09-01 19:01:48

It is possible and we do it to build images in airgapped envs. We do it for Photon & Ubuntu. As long as you have internal mirror repos for OS pkgs, Internally built/hosted k8s & friends pkgs. Let me know if you run into any specific issues.

:ty: Abhay Krishna Arunachalam
Abhay Krishna Arunachalam
2023-09-01 19:17:23

Thank you all for your valuable inputs. I'll reach out if I face any issues.

voor
2023-09-04 15:42:28

Historically, a few years ago we ran image builder in an AWS environment that was isolated from the Internet, unless there's been significant changes in external dependencies stuff should have had parameters for pointing to an internal equivalent

Vignesh Goutham
2023-09-12 21:37:53

@kiran keshavamurthy qq, can snap commands pull from private snap store proxy or other internal hosted methods? I was looking at this task and was wondering if we can provide a source for it.

kiran keshavamurthy
2023-09-12 21:48:38

Not sure about. We do not do any private snap registries.

Vignesh Goutham
2023-09-12 21:49:19

cool nw, thanks!

Danny Bessems
2023-09-01 10:30:20

What framework is used for tests within the image-builder repo? I have a need to set up testing of an OVA that we're building through Packer, and I want to see how other projects approach this.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-01 10:33:45

No frameworks, it's all custom bash scripts. This is also the reason that tests aren't as good as we'd like, there's a lot of functionality we haven't got tested because it's a lot of work and not clear to contributors how to add more.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-01 10:34:01

If you come across some nice frameworks for this we'd be very interested

Danny Bessems
2023-09-01 10:34:19

Yeah, asking around left and right 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-01 10:34:31

We are using for some of the validation

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-01 10:35:17

But I'll be honest, I'm far from an expert on this area so maybe some of the other contributors can comment more 🙂

Danny Bessems
2023-09-01 10:35:26

Yeah I did notice goss, looks like you combine it with the build itself though, and I want to test after a deployment of a completed ova 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-01 10:36:16

Yeah, I think it's used as validation during the build so if it doesn't match what's expected we fail the build rather than continuing to the end.

Danny Bessems
2023-09-01 10:37:07

I've used inspec in the same manner, but it doesn't cover testing deployment scenario's, so looking for something more elaborate.

Danny Bessems
2023-09-01 10:37:22

Anyway, thanks for the response, appreciated

jsturtevant
2023-09-01 16:44:56

goss is used to validate the image is created, in the image-builder CI, if the image builds properly and passes goss tests, we call it good.

Before we produce images for CAPZ, we do deploy the image in a cluster, to validate the cluster comes up.

Once it's a released image we have CI in CAPZ that runs kuberentes conformance tests a few times a day so we would catch any major regression there.

jsturtevant
2023-09-01 16:44:57

Other providers like AMI and GCE also run e2e tests in CI after images are produced so would catch regressions (we had one last week in GCE)

kiran keshavamurthy
2023-09-01 19:05:16

Same has what James said. We deploy k8s clusters with the OVAs and run install/upgrade/scaling/conformance etc tests. We have a internal testing framework in python to run other tests that’s needed.

Erkan Erol
2023-09-05 16:45:46

Hello. Is there anyone who builds OVA images for CAPV inside a container/pod? How do you install VMware tools to the container image?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-06 06:20:06

Is that the tools you’re referring to?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-06 06:21:48

Or do you mean you need the tools inside the container while building the image?

Erkan Erol
2023-09-06 08:44:04

I did mean the container that builds the node images. In image-builder project, vmware-iso builder is used. See

It has dependencies:

This VMware Packer builder is able to create VMware virtual machines from an ISO file as a source. It currently supports building virtual machines on hosts running VMware Fusion for OS X, VMware Workstation for Linux and Windows, and VMware Player on Linux. It can also build machines directly on VMware vSphere Hypervisor using SSH as opposed to the vSphere API.

I managed to install VMware Player as below:

FROM quay.io/giantswarm/capi-image-builder:1.6.8
USER root

# Check
RUN wget -q -O /tmp/VMWareWorkstation.bundle <br> && chmod +x /tmp/VMWareWorkstation.bundle <br> && /tmp/VMWareWorkstation.bundle --console --required --eulas-agreed <br> && rm -rf /tmp/**

USER imagebuilder
but it also relies on Virtual Infrastructure eXtension (VIX) SDK.

Erkan Erol
2023-09-06 08:45:02

I couldn’t find how to install this extension.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-06 09:48:06

Can you double check with the latest release of image-builder? If it’s still missing can you please open an issue? 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-06 09:50:28

Unless @kiran keshavamurthy happens to already know?

Erkan Erol
2023-09-06 10:29:57

Is <a href='http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.18'>registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.18</a> supposed to contain all necessary tools for all available makefile targets here?

Erkan Erol
2023-09-06 10:31:02

Just checked and the binaries (vmplayer, vmrun) don’t exist. I couldn’t find any code to install those into the image -builder container image too.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-06 10:31:09

In theory, yes. But there’s a lot to keep track of so things get missed.

Erkan Erol
2023-09-06 10:31:23

I see. Thanks for helping.

Erkan Erol
2023-09-06 10:45:17

Opened this issue to track

GitHub
:thx_thanks: Marcus Noble
Alessandro Giorgio Togna
2023-09-11 16:06:17

@Erkan Erol: I use VMWare Workstation on Linux to create the images with image-builder, since VMWare Player does not work; the command I use is make build-node-ova-local-ubuntu-2204

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-11 16:13:08

@Alessandro Giorgio Togna Do you use the Docker image or are you running the code directly?

Alessandro Giorgio Togna
2023-09-11 16:13:24

running the code directly

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-11 16:14:54

Yeah, Erkan is looking for how to run it within a container so that we can make use of it in our existing Tekton pipelines that builds our images for other platforms.

Alessandro Giorgio Togna
2023-09-11 16:15:40

I know, but I think you cannot use VMWare Player, you need the full Workstation

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-11 16:18:53

🤔 That's going to be problematic. 😞

Erkan Erol
2023-09-11 16:21:13

Yeah. We may think about using remote vSphere but not sure.

Alessandro Giorgio Togna
2023-09-11 16:21:15

this is from , and it does not mention VMware Player 😞

Erkan Erol
2023-09-11 16:23:02
I know, but I think you cannot use VMWare Player, you need the full Workstation
I am not fully sure but the packer builder doc says
This VMware Packer builder is able to create VMware virtual machines from an ISO file as a source. It currently supports building virtual machines on hosts running VMware Fusion for OS X, VMware Workstation for Linux and Windows, and VMware Player on Linux. It can also build machines directly on VMware vSphere Hypervisor using SSH as opposed to the vSphere API.
Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-11 16:23:42

I'll ask in the office hours in 10 min to see if anyone there knows an answer 🙂

Erkan Erol
2023-09-11 16:23:54

Thank you Marcus.

Alessandro Giorgio Togna
2023-09-11 16:23:54

packer might be able to do it, but image-builder does not...

Alessandro Giorgio Togna
2023-09-11 16:24:10

I tried using it to no avail

Alessandro Giorgio Togna
2023-09-11 16:24:19

I had to switch to Workstation

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-11 16:24:28

image-builder is just an opinionated wrapper around packer. If you know how to do it with Packer we can update this project to support it. (Hopefully)

👍 Alessandro Giorgio Togna
Erkan Erol
2023-09-11 16:31:57

This was my assumption too 🙂 I managed to install VMWare Player but also the packer builder requires vmrun command, which is a part of Virtual Infrastructure eXtension (VIX) SDK according to doc. See

Erkan Erol
2023-09-12 11:40:28

@Marcus Noble Could you learn anything yesterday? Was there anyone who has an answer?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-12 11:41:34

Sorry, forgot to report back. I’m afraid not. The attendance was very little and none with CAPV experience. 😔 Did you manage to get any response from the CAPV team?

Erkan Erol
2023-09-12 11:52:23

Nope. Will push it again.

kiran keshavamurthy
2023-09-12 21:51:25

Hello. Is there anyone who builds OVA images for CAPV inside a container/pod? How do you install VMware tools to the container image?
When building from the container we recommend using vSphere builders.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-13 07:11:38

Are you doing that with image-builder? I thought we only had the OVA vSphere targets? (I may be misunderstanding though)

Erkan Erol
2023-09-13 09:10:43

From the images/capi directory, run make build-node-ova--, where is your target hypervisor (local or vsphere) and is the desired operating system. The available choices are listed via make help.
It is possible to use a remote vSphere to build images.

Erkan Erol
2023-09-20 10:52:17

I updated the upstream issue . Shortly, I managed to install VMware tools for vmware-iso builder but there are still some issues.

As far as I understand from people’s commens, make build-node-ova-local-flatcar is not being used&tested actively. If so, I would delete it from the repo.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-20 11:03:46

There are several targets not being tested due to not having infrastructure to test them on. Its a known issue and something we're hoping to improve but without access to the relevant cloud infrastructure there's not much we're able to do other than rely on contributors and users to test and report issues. 😞

Erkan Erol
2023-09-20 17:09:21

Update: make build-node-ova-vsphere-flatcar worked in my first attempt. Many people reported they use vSphere here. It seems it is a defacto standard.

Vignesh Goutham
2023-09-08 22:15:20

Could I please get a review on this fix for raw builds? gratitude-thank-you

Vignesh Goutham
2023-09-13 19:09:13

@Marcus Noble /@mboersma could you folks take a pass at this

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-13 19:12:03

👍 Sorry bout that.
lgtm approved

Vignesh Goutham
2023-09-13 19:14:58

nw, thanks!

Slackbot
2023-09-11 15:30:25

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Pengfei Huang
2023-09-11 22:22:12

@Pengfei Huang has joined the channel

mboersma
2023-09-13 17:28:26

Now that we've merged the pkgs.k8s.io change, I think we should do an image-builder v0.1.19 release (before we update to containerd 1.7). I'm happy to do a release today unless there are reasons to wait or other objections.

✅ Drew Hudson-Viles, jsturtevant, Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-13 18:39:06

I meant to ask this in the last office hours and forgot:

The CFP for the Kubernetes Contributor Summit in Chicago closes on Friday. Do we want to submit a session for image-builder at all?

Would anyone be interested in either:

  • a "current state of image-builder" type presentation

  • a more general meeting to discuss the future of the project and work on building out some sort of roadmap?

Or maybe something else entirely?

Drew Hudson-Viles
2023-09-13 18:42:38

I love the idea but I doubt I could swing it to get to Chicago just yet 😛. I was going to try and get something for KCD in London (which I see you're at) but too much on my plate meant I missed the deadline even though my boss pestered me (he's one of the organisers).

All that being said I think a state of the project would be a good one as I do wonder sometimes if some people even know about it. In my previous job we never used it and it would have made life simpler if we'd known about it.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-13 18:44:58

Yeah, it does kinda depend on who will be at KubeCon NA or not as to if it'll be useful.

mboersma
2023-09-13 19:46:05

I probably can't make it to Chicago, but thumbs up to a session, maybe we could participate remotely.

mboersma
2023-09-13 19:44:44

announce Image-builder v0.1.19 is now available:
Thanks to all contributors!

🎉 Marcus Noble, Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-14 12:12:26

We should add a Breaking Changes section to the release notes to indicate that people might need to change the kubernetesdebversion variable if they're providing it themselves. (see this thread - )

Maximilian Rink (https://kubernetes.slack.com/team/ULXC4BQFL)
Danny Bessems
2023-09-14 09:02:33

Anyone ever noticed that ntp settings are not picked up properly in Ubuntu2204 vSphere OVA, even though upon deployment guestinfo.userdata does have ntp.enabled: true and ntp.servers: [] set?

nikparasyr
2023-09-14 09:08:41

Danny Bessems (https://kubernetes.slack.com/team/UQU1LL28L)
nikparasyr
2023-09-14 09:09:29

oh wait. that is you asking it 😛

Danny Bessems
2023-09-14 09:09:43

Yeah, but thanks for pointing it out, I missed that last reply 😄

Danny Bessems
2023-09-14 09:12:15

So it's not really a image-builder issue, but a cloud-init issue?

nikparasyr
2023-09-15 07:16:01

i'd say so. but image-builder allows you to add your custom role when building an image. so if you know that all your clusters will have the same ntp config, then you can build it in the image and not rely on cloudinit

Danny Bessems
2023-09-15 08:29:07

Unfortunately that's very much not the case for me, but also, it seems the issue has gone away with a new run of image-builder :)

Danny Bessems
2023-09-15 08:29:26

Thanks for responding though, much appreciated!

👍 nikparasyr
Maximilian Rink
2023-09-14 11:56:17

hmm, was anyone else able to successfully build images with the last release (1.26.9 etc.)?
Im unable to install the debs for the new releases

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-14 12:01:23

Which provider and OS?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-14 12:01:36

If GCE, it might be now fixed with this PR that's just merged -

GitHub
Maximilian Rink
2023-09-14 12:02:00

ubuntu, and ova / raw/qemu

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-14 12:03:07

Oh. Hmmm... maybe that has the same problem as GCE then? Are you able to try overwriting the kubernetesdebversion to 1.24.15-1.1 to see if that works?

Maximilian Rink
2023-09-14 12:03:28

but yeah, might be idential as the build version is still the old one

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-14 12:03:38

/cc @mboersma FYI ☝️

😮 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-14 12:07:02

Oh actually, I don't think it is that. The GCE one was because it was an override that wasn't updated during the original change. So unless you're providing the kubernetesdebversion variable yourself it should be working unless we've missed something. 😞

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-14 12:07:20

Do you get an error message?

Maximilian Rink
2023-09-14 12:08:19

    vsphere-iso.vsphere: fatal: [default]: FAILED! => {"cache_update_time": 1694680197, "cache_updated": false, "changed": false, "msg": "no available installation candidate for kubelet=1.24.17-00"}
Yeah, but i guess since the release yesterday the old -00 postfix needs to be changed and i missed that

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-14 12:10:32

Ah yeah. I guess we need to announce that more clearly. I didn't think about people setting the values themselves. 🤦‍♂️

Maximilian Rink
2023-09-14 12:42:35

So yeah, after fixing the version it worked

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-14 13:05:36

Thanks for confirming 🙂

mboersma
2023-09-14 15:59:36

Sorry, I didn't realize there were overrides of the kubernetesdebversion elsewhere, should have caught that in the PR.

Maximilian Rink
2023-09-14 16:12:35

@mboersma it was in our overrides that we have locally to build our images, ive missed that those need migrating

c
2023-09-19 17:44:15

@c has joined the channel

Fredrik Björkman
2023-09-22 06:29:08

@Fredrik Björkman has left the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-22 15:29:42

@mboersma I can't seem to get the ensure-ansible-lint.sh working on my Mac 😞
ensurepy3bin ansible-lint keeps failing for me and I'm not sure why. It look like it is installed (as pip3 show ansible-lint works) but I don't have the cli available anywhere 😕

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-22 15:30:54

oh actually, looks like my PATH might not be right 🤔

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-22 15:32:04

Yup! User error 🤦‍♂️ Ignore me. 😆

Drew Hudson-Viles
2023-09-22 15:41:03

I'm sure it won't be the last time 😄 I did exactly that the other day and wondered why ansible wasn't working when it was clearly installed facepalm-1720

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-22 15:41:45

It's been a long day 😅

😄 Drew Hudson-Viles
❤️ mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-25 12:57:17

📣 There's currently no topics on the agenda for todays office hours. If no one adds anything in the next few hours I'm going to suggest that we cancel the sync for this week. 🙂

👍 Drew Hudson-Viles
lilac
2023-09-25 14:32:06

@lilac has joined the channel

Slackbot
2023-09-25 15:30:02

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-25 15:34:40

☝️ Agenda still empty so I'm cancelling the office hours for this week 🙂 See y'all in 2 weeks!

👍 jsturtevant
Drew Hudson-Viles
2023-09-25 15:35:06

Makes sense think3d

mboersma
2023-09-25 15:43:56

Ok with me, I didn't have anything in particular to discuss. Until next time!

Ashutosh
2023-09-26 09:05:42

@Ashutosh has joined the channel

Ashutosh
2023-09-26 09:06:51

I was wondering how I could build RHCOS images using image builder as Red Hat uses RHCOS for the compute nodes.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-09-26 09:11:36

We don't currently have any support for RedHat coreos in image-builder. If you'd be willing to work on a PR to introduce it we'd be very welcoming to that. It would require an understanding of Packer to build the images.

I also wasn't aware that RedHat had their own CAPI provider. Or have I misunderstood your use?

Ashutosh
2023-09-26 09:43:29

@Christophe Jauffret Copying in Christophe

Ashutosh
2023-09-26 09:59:53

It looks like its in Technology preview feature which will only support AWS and Azure for now. So Nutanix is not planned by them. Because we are using the Red Hat subscriptions, I guess we will have to wait till it is officially supported by Red Hat.

Christophe Jauffret
2023-09-26 12:04:39

Hello @Ashutosh, what is your goal ?

let’s first resume some few points:

  • image-builder project has goal to build upstream CAPI ready image

  • image-builder use packer as main tools

  • we (Nutanix) have a packer plugin for our platform

  • we are maintaining a Nutanix flavor for image-builder based on our Packer plugin

  • image-builder doesn’t suppport RHCOS, but on other side RHCOS is initially thought to work with CRI-O and Openshift, so not really the target of image-builder/capi

  • Concerning CAPI , we have a Nutanix Infrastructure Provider for CAPI (called CAPX) but it take care only of infrastructure , boostrap is managed by different CAPI provider and not specific to the infrastructure

  • If your goal is to try to use RHCOS in a CAPI context, there is an alpha support for ignition in kubeadm bootstrap provider

  • If your goal is to use RHCOS as VM OS , you can pass ignition file through Nutanix Cloud-init support.

  • Openshift IPI installer is based on an old fork of MAPI/CAPI project and don’t use directly upstream CAPI infra provider

  • In Openshift , Nutanix Images are build/provided by RedHat

  • There is a plan (I let RedHat confirm) to move IPI to upstream CAPI in the future and this is what you see as TP in 4.14


Don’t hesitate to explain exactly what you try to achieve, to see how we can help you.

Ashutosh
2023-09-26 12:16:32

Hello @Christophe Jauffret Thank you for your comment. My goal is to automate the creation of new clusters and reduce the human errors that could happen when using the IPI method of installing a cluster. Today with IPI every cluster has to be installed manually and this is error prone. Cluster API will not just make the installation of new clusters less cumbersome but also updating and upgrading these clusters in a standardised way. So my goal is to use RHCOS in a CAPI context. But doesn’t the Openshift IPI automatically provision VMs in Nutanix with RHCO? So under I was wondering why RHCO wasn’t listed although RHEL is listed. My goal was to test this using an openshift cluster installed on Nutanix which will act as the management cluster and then spin up a new cluster using CAPI.

esierra
2023-09-26 12:11:29

@esierra has joined the channel

fad3t
2023-09-26 13:27:33

@fad3t has joined the channel

mboersma
2023-09-27 19:06:40

green-light-alert There are a couple of good-first-issue items in image-builder. If you're looking for an easy way to get involved in the project, we would love your help!

💙 Marcus Noble, Erkan Erol, RAKESH BOINAPALLY
RAKESH BOINAPALLY
2023-09-29 17:05:38

I would like to get started on the linting one

RAKESH BOINAPALLY
2023-09-29 17:05:41

if that's open

mboersma
2023-09-29 18:52:02

They've both been assigned. I'll try to create some more issues we need help with, and if we don't have a PR for the first one soon we can reassign it.

👍 RAKESH BOINAPALLY
RAKESH BOINAPALLY
2023-09-29 19:14:44

sure thing

RAKESH BOINAPALLY
2023-10-05 03:31:50

@mboersma just wondering if you can create few more issues

cecile
2023-09-27 19:22:31

Huge thank you for filling out the SIG survey thank-you

:kubernetes: mboersma
mboersma
2023-09-27 19:37:14

Thanks especially to @Marcus Noble for making sure it got done. 🙂

:ty: jsturtevant
ciomaire
2023-09-27 21:14:30

@ciomaire has joined the channel

faithkovi
2023-09-28 00:02:50

@faithkovi has joined the channel

Shiva Abhishek
2023-09-28 07:39:34

@Shiva Abhishek has joined the channel

Erkan Erol
2023-09-28 09:53:35

Hello. I opened a PR for vSphere builder to expose ssh_proxy variables.

Is there a transitive way to set any variables for any builders without exposing them as I do in the PR?

mboersma
2023-09-29 14:15:11

Thanks for the PR! I commented there.

mboersma
2023-09-29 14:15:43

tl:dr I don't know of another way to do it, but I'm far from a Packer expert, so we should see if other image-builder folks have ideas.

Erkan Erol
2023-10-02 09:15:18

Thanks a lot for taking care of it!

Ankriti Sachan
2023-09-28 17:32:33

@Ankriti Sachan has joined the channel

RAKESH BOINAPALLY
2023-09-29 17:05:13

@RAKESH BOINAPALLY has joined the channel

Sambhav Gupta
2023-09-30 17:00:29

@Sambhav Gupta has joined the channel

Jey
2023-10-02 18:52:00

@Jey has joined the channel

Shashinandan Srinivasa
2023-10-02 21:07:56

@Shashinandan Srinivasa has joined the channel

Shashinandan Srinivasa
2023-10-02 21:51:24

Hello folks, are you guys able to build ubuntu 22.04 for vsphere? I see its "waiting for ssh" on the vSphere VM.. In the console of VM, its either asking for manual inputs starting from selecting keyboard or like one below

Shashinandan Srinivasa
2023-10-02 21:52:53

Image Builder Releases

The current release of Image Builder is [v0.1.19][] (September 13, 2023). The corresponding container image is <a href='http://registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.19'>registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.19</a>.

## Release Process

For more detail about image-builder project releases, see the [Image Builder Book][].


[v0.1.19]:
[Image Builder Book]:

GitHub
feitnomore
2023-10-03 21:33:06

I am trying to add Ubuntu 23.04 to qemu, but I am failing miserably

fad3t
2023-10-04 07:03:52

maybe you can share some more details, it's hard to help just knowing it fails 😛

feitnomore
2023-10-04 11:20:09

I could achieve it

feitnomore
2023-10-04 11:20:20

it was a problem with apt, that was trying to use a local mirror

feitnomore
2023-10-04 11:20:28

I need to test the images generated though

feitnomore
2023-10-04 11:23:02

apt:
mirror-selection:
primary:
- uri:

feitnomore
2023-10-04 11:23:15

I've added this to my autoinstall cloud-config

feitnomore
2023-10-03 21:33:09

can anyone assist?

ARYAN
2023-10-04 15:55:44

@ARYAN has joined the channel

Alessandro Giorgio Togna
2023-10-04 18:40:32

I am trying to rebuild a CAPI image for k8s 1.26.6 using this values:
{
"kubernetessemver": "v1.26.6",
"kubernetes
debversion": "1.26.6-00",
}
however the make is failing since it cannot find kubectl/kubeadm 1.26.6-00.
Where do I get the list of available values for kubernetes
deb_version ?

Drew Hudson-Viles
2023-10-04 18:46:05

Try apt-cache madison kubeadm on a system where the repo available. This will yield the versions available.

Alessandro Giorgio Togna
2023-10-04 18:48:43

if I do that the 1.26.6-00 version is available, but make fails with a "cannot find the 1.26.6-00" version of the package

Alessandro Giorgio Togna
2023-10-04 18:49:44

for instance, the default version now is:
"kubernetesdebversion": "1.26.7-1.1"
which I cannot find with apt-cache

Drew Hudson-Viles
2023-10-04 18:50:37

Are you using the new apt repos on the machine on which you're running apt-cache?

Alessandro Giorgio Togna
2023-10-04 18:52:03

maybe not, I'll check now

Drew Hudson-Viles
2023-10-04 18:52:41

Cool 🙂 I suspect it may be the change in repos which is why you're not seeing it. Let me know if that doesn't work.

🙌 Alessandro Giorgio Togna
Alessandro Giorgio Togna
2023-10-04 18:56:06

thank you, I had the "wrong" repo where all the versions are -00

Alessandro Giorgio Togna
2023-10-04 18:56:32

I configured the new repo (and all versions are -1.1)

Drew Hudson-Viles
2023-10-04 18:56:40

Glad that's worked 🙂. It was a fairly recent change so could be easily missed.

Animesh Pandey
2023-10-05 05:33:27

@Animesh Pandey has joined the channel

aniruddha
2023-10-05 19:27:15

@aniruddha has left the channel

feitnomore
2023-10-06 20:13:18

I could build the Ubuntu 23.04 image, however, looks like cloud-init is not working well... no dhcp, no user created, no kubeadm

feitnomore
2023-10-06 20:13:42

I could log into the machine by creating a user by hand, if I run dhclient it gets ip

feitnomore
2023-10-06 20:13:55

hostname is not changed as well, it is localhost.localdomain

feitnomore
2023-10-06 20:14:18

I could mount the cloud-init disk /dev/vda, and I see it has a openstack dir with everything inside it

feitnomore
2023-10-07 00:23:41

(I am trying to build a qemu one)

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-09 13:04:36

We've got the image-builder office hours later today but currently the agenda is empty. I'm not aware of any topics that need discussing and if there's nothing added by 1 hour before the call I plan to cancel it and meet up in a couple weeks for the next sync instead.

✅ Drew Hudson-Viles
Slackbot
2023-10-09 15:30:14

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-09 15:37:26

Agenda empty. Cancelling office hours 🙂 See y'all in 2 weeks!

👍 mboersma, jsturtevant
mboersma
2023-10-11 17:21:34

Image-builder CI is currently broken due to a python update in the e2e environment. I think this PR fixes it:

:approval: Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-12 08:28:38

I asked this on the PR but prob best to chat here...

Do you happen to know if the E2E environment changes were announced anywhere? Is there something we should be keeping an eye on for potential problems?

Arnaud (he/him)
2023-10-12 04:59:25

@Arnaud (he/him) has left the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-13 16:40:00

We've not done a release for the past month - anyone against doing a new one? Any changes people would like to get in before we do?
Changes pending release:

Drew Hudson-Viles
2023-10-13 16:40:57

Seems like a good idea to me

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-13 16:41:56

It's end of day for me and I'm not in a rush to have it done right now so I might give it a little time (e.g. until Monday) for people to speak up with any changes they want to get in.

Drew Hudson-Viles
2023-10-13 16:42:23

I don'yeah makes sense. No one wants a Friday release 😁

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-13 16:43:11

Right?! I have a habit of breaking clusters on Friday 😅

😆 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-16 07:58:47

I'm going to start the process of getting v0.1.20 released

🙌 Abhay Krishna Arunachalam
:ty: mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-16 08:23:34

Promotion PR -

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-16 08:27:13

And docs updates:

🙌 Drew Hudson-Viles
Siddhiprada Mohapatro
2023-10-14 09:05:31

@Siddhiprada Mohapatro has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-16 13:10:55

Image-builder v0.1.20 is now available:
Thanks to all contributors!

GitHub
🎉 mboersma, RAKESH BOINAPALLY, Anurag
Amim Knabben
2023-10-16 18:24:47

hey folks anyone else having issues with ansible-galaxy?

Amim Knabben
2023-10-16 18:26:24

  41 ~ │ ansible-galaxy -vvv collection install <br>  42   │   community.general <br>  43   │   ansible.posix <br>  44   │   'ansible.windows:>=1.7.0' <br>  45   │   community.windows
community.general is failing for me when running hack/ensure-ansible.sh
ERROR! Unexpected Exception, this is probably a bug: '/api/v3/plugin/ansible/content/published/collections/index/community/general/versions/'

with the 2.11.5 the workaround is to add on ansible.cfg

Amim Knabben
2023-10-16 18:29:34

GitHub
Amim Knabben
2023-10-16 18:29:50

should ansible be bumped instead?

Amim Knabben
2023-10-16 19:07:16

oh i'm using an old version, seems the latest is 2.15 already

mboersma
2023-10-16 19:07:41

Oops sorry, I meant to reply in this thread.

The problem I ran into is this, not sure if that's exactly the same as yours:


But image-builder does effectively require ansible 2.15.5 now, although the script doesn't check versions, just that ansible exists.

👍 Amim Knabben
mboersma
2023-10-16 19:08:35

Maybe we should open an issue to change the ensure-ansible.sh script to also check the version.

Amim Knabben
2023-10-16 19:09:33

yea maybe a warning

Amim Knabben
2023-10-16 19:09:53

the Unexpected Exception without -vvv is not useful at all

Amim Knabben
2023-10-16 19:09:56

let me open it

Amim Knabben
2023-10-16 19:17:23

GitHub
mboersma
2023-10-16 19:20:43

Thank you!

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-19 08:35:30

I don't think we want to use the old-galaxy do we? The solution we want for image-builder is version checking with a user-friendly error if found to be less than 2.13.9 right?

Amim Knabben
2023-10-19 11:00:36

@Marcus Noble the problem is breaking all the

Amim Knabben
2023-10-19 11:00:49

but forward I think it makes sense to fail close

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-19 11:02:33

Oh sorry, this is an external change that is now causing issues?

Amim Knabben
2023-10-19 13:14:41

yes, its an ansible server

Travis Holton
2023-10-17 02:41:26

@Travis Holton has joined the channel

feitnomore
2023-10-17 13:50:44

I
am using Cluster-API with Kubevirt and Kubeadm... question: Is there
any documentation on what exactly the image needs to look like to work
on Cluster API? I've built a few images with image-builder that are not
working and I am not sure how to troubleshoot.... I am not sure if there is a package missing, a service that was not
started, or a config file... Is there any document about the
requirements on the image?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-19 08:36:56

Hey Marcelo, it might be useful to explain what error you're seeing and what vars / make target you're using, etc.
I don't have any experience with Kubevirt so might not be able to help you myself but might be able to point you in the right direction.

feitnomore
2023-10-19 10:53:41

Any tips are pretty much appreciated

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-19 10:56:36

I can't help without the information I asked for.

  • What error are you seeing?

  • What make target are you running?

  • What variables are you providing (if any)?


Also, would be good to know if you're using the latest released version and if you're running it directly or with the container image.

feitnomore
2023-10-19 11:02:12

ok, haven't seem those bullet points, sorry

feitnomore
2023-10-19 11:03:04

let's go with the easier one... I've copied ubuntu 22.04 to 23.04, and I am running qemu-ubuntu for it

feitnomore
2023-10-19 11:03:17

the image builds ok

feitnomore
2023-10-19 11:03:32

but kubeadm doesn't bootstrap on it, the control plane doesn't get up

feitnomore
2023-10-19 11:03:38

I am not sure where to look for errors

feitnomore
2023-10-19 11:04:23
export PACKER_FLAGS="--var 'kubernetes_rpm_version=1.27.6-0' --var 'kubernetes_semver=v1.27.6' --var 'kubernetes_series=v1.27'  --var 'kubernetes_deb_version=1.27.6-00' --var 'disk-size=6144'"
Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-19 11:04:39

Oh right, so you're trying to build a new OS that we don't yet support?

feitnomore
2023-10-19 11:05:07

I think that is the easy one... I've used the same flags, to build-qemu-flatcar

feitnomore
2023-10-19 11:05:18

the image boots, however, I never get the cluster up

feitnomore
2023-10-19 11:05:33

I can troubleshoot both... those are the ones that I've been trying so far

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-19 11:07:55

Gotcha!
Ok, so hard to say what could be going on. If you haven't done so already I recommend taking a look at the boot log to see if there are any failures that stand out in there (e.g. failure to pull something from the internet or start a service). If that all looks ok I'd suggest taking a look at journalctl -f and look out for services (specifically kubeadm related) that are failing to start.

Did the build log output have any warning that might be related?

feitnomore
2023-10-19 11:16:50

I haven't seen

feitnomore
2023-10-19 11:22:45

my best clue, when it comes to ubuntu 23.04 is that it might have something to do with the fact that the NIC name changes... its not in enps, its like ens**

feitnomore
2023-10-19 11:23:08

and I see there is a netplan file the image-build uploads over there, that I think might be related to the problem

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-19 12:07:26
feitnomore
2023-10-19 14:50:18

generating a new image now

feitnomore
2023-10-19 15:24:39

Pastebin
feitnomore
2023-10-19 15:24:44

now I'll try it

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-19 15:52:49

Did that run successfully then?

feitnomore
2023-10-19 15:54:11

looks like 🙂

feitnomore
2023-10-19 15:54:30

my system is a little bit complex, but I'm importing the image right now into my KubeVirt Cluster to test

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-19 15:56:14

🤞

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-19 15:56:35

🤔 Need to have a think about how best to handle this for different OS's

feitnomore
2023-10-19 18:32:24

I think after some troubleshooting that I could get it working

feitnomore
2023-10-19 18:32:51

control plane is up, waiting for worker nodes

feitnomore
2023-10-19 18:36:15

gotta figure out what went wrong in the build process, to have it fixed

feitnomore
2023-10-19 18:36:40

but I think the trick was done by running sudo cloud-init clean --machine-id

feitnomore
2023-10-19 18:36:53

and systemctl enable cloud-init

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-19 18:48:02

Glad to hear you're making progress! 😄

feitnomore
2023-10-19 18:50:08

I might have done a lot of mess over here, but its working

😆 Marcus Noble
feitnomore
2023-10-19 18:50:10

lol

feitnomore
2023-10-19 19:34:45

so, for now, I have a guess it has something to do with the cloud-init clean, and not the netplan file

feitnomore
2023-10-19 19:35:04

cause all my other images work, and they don't have the netplan file, as I'm building as a plain simple qemu image

feitnomore
2023-10-19 19:35:15

and I am not setting kubevirt=true

feitnomore
2023-10-17 13:50:59

Example: flatcar isn't working

Alex B.
2023-10-19 13:38:14

@Alex B. has joined the channel

discostu
2023-10-20 07:23:54

@discostu has joined the channel

Slackbot
2023-10-23 15:30:19

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-23 15:33:06

☝️ Any topics for today?

mboersma
2023-10-23 16:02:32

Nothing in particular from me.

mboersma
2023-10-23 16:02:54

But I'm happy to meet if anyone has a discussion topic. 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-23 16:15:22

👍 I'm about so can join and see if there's anything to discuss 🙂

👍 mboersma
Shalin Patel
2023-10-23 17:33:37

@Shalin Patel has joined the channel

Yike Wang
2023-10-25 14:50:18

when building 1.28.2 image based on v0.1.20 release, I meet:

    amazon-ebs.{{user build&#95;name}}: TASK [include_role : kubernetes] *
amazon-ebs.{{user build&#95;name}}:
amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Add the Kubernetes repo key] *

amazon-ebs.{{user build&#95;name}}: changed: [default]
amazon-ebs.{{user build&#95;name}}:
amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Add the Kubernetes repo] *
amazon-ebs.{{user build&#95;name}}: changed: [default]
amazon-ebs.{{user build&#95;name}}:
amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Install Kubernetes]
*
amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": false, "msg": "No package matching 'kubelet' is available"}
Any idea? Should I change some registry source or something, or downgrade something? Thanks!

Yike Wang
2023-10-25 14:50:58

my config file:

 /Users/yikew/Working/capa/image-builder/1.28.2/config.json
{
"kubernetes_series": "1.28",
"kubernetes_semver": "v1.28.2",
"kubernetes_rpm_version": "1.28.2-0",
"kubernetes_deb_version": "1.28.2-00",
"kubernetes_source_type": "pkg",
"kubernetes_http_source": "",
"kubernetes_rpm_repo": "",
"kubernetes_rpm_gpg_key": "\" \"",
"kubernetes_rpm_gpg_check": "True",
"kubernetes_deb_repo": "\" kubernetes-xenial\"",
"kubernetes_deb_gpg_key": "",
"kubernetes_container_registry": "registry.k8s.io",
"kubernetes_load_additional_imgs": "false",
"kubeadm_template": "etc/kubeadm.yml",
"containerd_version": "1.7.6",
"containerd_sha256": "20da1f2252d2033594b06e1eb68dd4906ff439f83f1003b7ebacdffcb4b95bdc"
}

Yike Wang
2023-10-25 15:10:11

it happens when building build-ami-ubuntu-2004 and build-ami-ubuntu-2204
build-ami-centos-7 and build-ami-amazon-2 work fine.

Yike Wang
2023-10-25 16:02:19

downgrade image-builder to v0.1.17 and then it works on build-ami-ubuntu-2004 and build-ami-ubuntu-2204 finally.

mboersma
2023-10-25 17:36:39

@Yike Wang I think this is because the format of kubernetesdebversion and kubernetesrpmversion changed slightly in .

The packages now come from the approved pkgs.k8s.io repository, but Kubernetes is using new tooling to publish. So I think this will work with image-builder v0.1.20:

"kubernetes_rpm_version": "1.28.2",
"kubernetes_deb_version": "1.28.2-1.1",

👍 Drew Hudson-Viles
Yike Wang
2023-10-28 15:56:01

I follow the new configs in . , but I'll hit:

    amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": false, "msg": "Failed to download key at : HTTP Error 403: Forbidden"}
my config:
{
"kubernetes_series": "1.28",
"kubernetes_semver": "v1.28.3",
"kubernetes_rpm_version": "1.28.3",
"kubernetes_deb_version": "1.28.3-1.1",
"kubernetes_source_type": "pkg",
"kubernetes_http_source": "",
"kubernetes_rpm_repo": " user kubernetes&#95;series }}/rpm/",
"kubernetes_rpm_gpg_key": " user kubernetes&#95;series }}/rpm/repodata/repomd.xml.key",
"kubernetes_rpm_gpg_check": "True",
"kubernetes_deb_repo": " user kubernetes&#95;series }}/deb/",
"kubernetes_deb_gpg_key": " user kubernetes&#95;series }}/deb/Release.key",
"kubernetes_container_registry": "registry.k8s.io",
"kubernetes_load_additional_imgs": "false",
"kubeadm_template": "etc/kubeadm.yml",
"containerd_version": "1.7.6",
"containerd_sha256": "20da1f2252d2033594b06e1eb68dd4906ff439f83f1003b7ebacdffcb4b95bdc"
}
do you have idea

Abhay Krishna Arunachalam
2023-10-28 21:52:49

Try changing the kubernetes_series parameter to v1.28, with the leading v. That should fix it.

curl -ILs -o /dev/null -w "%{http_code}" https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/1.28/deb/Release.key
403

curl -ILs -o /dev/null -w "%{http_code}" https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.28/deb/Release.key
200

Yike Wang
2023-10-29 02:37:27

I always use 1.28 as kubernetesseries without problem, never notice it. Thank you! @Abhay Krishna Arunachalam it works!

But for amazon image no
proxy=* make build-ami-amazon-2 , there is no satisfied cri-tools found in the new repositories

amazon-ebs.{{user build&#95;name}}: TASK [kubernetes : Install Kubernetes] *
amazon-ebs.{{user build&#95;name}}: fatal: [default]: FAILED! => {"changed": false, "changes": {"installed": ["kubelet-1.28.3", "kubeadm-1.28.3", "kubectl-1.28.3", "kubernetes-cni-1.2.0"]}, "msg": "Error: Package: kubeadm-1.28.3-150500.1.1.x86_64 (kubernetes)\n Requires: cri-tools >= 1.28.0\n Available: cri-tools-1.25.0-1.amzn2.0.1.x86_64 (amzn2-core)\n cri-tools = 1.25.0-1.amzn2.0.1\n Available: cri-tools-1.26.1-1.amzn2.0.1.x86_64 (amzn2-core)\n cri-tools = 1.26.1-1.amzn2.0.1\n Available: cri-tools-1.26.1-1.amzn2.0.2.x86_64 (amzn2-core)\n cri-tools = 1.26.1-1.amzn2.0.2\n", "rc": 1, "results": ["Loaded plugins: extras_suggestions, langpacks, priorities, update-motd\n227 packages excluded due to repository priority protections\nResolving Dependencies\n--> Running transaction check\n---> Package kubeadm.x86_64 0:1.28.3-150500.1.1 will be installed\n--> Processing Dependency: cri-tools >= 1.28.0 for package: kubeadm-1.28.3-150500.1.1.x86_64\n---> Package kubectl.x86_64 0:1.28.3-150500.1.1 will be installed\n---> Package kubelet.x86_64 0:1.28.3-150500.1.1 will be installed\n---> Package kubernetes-cni.x86_64 0:1.2.0-150500.2.1 will be installed\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"]}
Should I file an issue somewhere?

feitnomore
2023-10-25 18:56:27

I've got PR opened

GitHub
feitnomore
2023-10-25 18:56:41

if anyone wants to talk about it, feel free to ping me

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-27 08:10:26

Is there anyone with an understanding / experience with cloud-init in Ubuntu that might be able to help out @Shalin Patel with this issues - ? It looks like something changed in the 23.3.1-0ubuntu1~20.04.1 release of cloud-init that broke builds for cluster-api-provider-aws. 😞

GitHub
Drew Hudson-Viles
2023-10-27 09:17:08

I'm not familiar enough with it to be able to say what's changed to be honest. If I get chance this weekend I'll have a look but it does look like something in the package has changed to cause this since they can downgrade and get it working.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-27 09:17:54

Yeah exactly. But I've also no clue about that package so was hoping to get some insight from the hive mind of Slack 😄

Shalin Patel
2023-10-27 17:09:57

Thank you for bringing it to attention. This issue was reported and observed by us in CAPA. CAPA members are looking into it.

Robert Van Voorhees (https://kubernetes.slack.com/team/U2B8TBG58)
Erkan Erol
2023-10-31 12:07:44

I built an OVA image (flatcar-stable-3602.2.1-kube-v1.24.12) and tried to import it to vSphere. I hit this issue

Issues detected with selected template. Details: - 51:7:VALUE_ILLEGAL: Value ''VirtualSCSI'' of ResourceSubType element not found in [lsilogic, lsilogicsas]. - 94:7:VALUE_ILLEGAL: Value ''3'' of Parent element does not refer to a ref of type DiskControllerReference.

When I compare the OVF file in my image(flatcar-stable-3602.2.1-kube-v1.24.12) and the current upstream image (flatcar-stable-3374.2.5-kube-v1.24.11), there is no diff related to VirtualSCSI

However, I found this diff.
# upstream image


# my image

I manually change this and tried again. It worked. Then I check the repo and found this change https://github.com/kubernetes-sigs/image-builder/commit/d410869b84532819f5299a77188c4af20b5d283d#diff-e3ec89888eeec052751[…]e01561d14c5ffd22fb677a2d

I tested upstream 1.28.0 that has the change. It failed too. I manually changed the osType and it worked again. Is there anyone who is aware of this issue?

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-31 15:24:19

This is the PR that introduced the change:

@Yiyi Zhou as the author of that PR are you able to offer any insight?

@Erkan Erol Can you confirm what version of vSphere you're working with? If I'm understanding this page correctly it looks like otherLinux64Guest was introduced in v5.0. But I'm just trying to search for related things and don't have much actual insight into vSphere stuff as you know 😅

GitHub
Yiyi Zhou
2023-10-31 17:45:35

Yeah otherLinux64Guest is introduced since 5.0. If this change is breaking backward compatibility, I will file to revert.

Erkan Erol
2023-11-02 08:44:45

The version of vSphere I use is 7.0.3 and it gives the weird error I mentioned above for the images that contains otherLinux64Guest. Interesting. @Yiyi Zhou Have you ever tried to upload to vSphere an image after this change? What could I be missing?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-02 08:46:51

Weird 😕 I guess if the linux-64 one works I think you can set vsphereguestos_type in your provided vars to override it. I haven't actually checked through the code but that PR seems to suggest that would work. 🙂

Erkan Erol
2023-11-02 08:51:23

Really? Let me check.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-02 08:52:00

I think so. This line seems to suggest so:

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-02 08:52:31

Actually, let me double check to be sure. 1 min.

Erkan Erol
2023-11-02 08:53:50

I didn’t see anything in the python script that respects this variable.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-02 08:53:53

ok, things get stranger. It looks like flatcar should be other3xLinux64Guest based on this:

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-02 08:54:34

Which python script? It looks to be passed to Packer as guestostype in packer-node.json

Erkan Erol
2023-11-02 08:56:28

images/capi/hack/image-build-ova.py overrides OVF file based on this template

Erkan Erol
2023-11-02 08:57:06

It has a hardcoded dictionary here

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-02 08:58:25

Yeah just looking at that and I'm now even more confused 😕 Do you know where that builddata comes from? For Flatcar I would have expected it to be other3xLinux64Guest based on the link I posted above but there's no key in the map for that so I would expect it to end up with an empty string for the OSTYPE

Erkan Erol
2023-11-02 09:00:27

It is the custom_data in the packer-manifest.json

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-02 09:00:41

Oh, I'm starting to understand I think.
In this file:
guestostype is set in multiple places and using two different values, either user guest_os_type or user vsphere_guest_os_type
I don't know if that is correct.

Erkan Erol
2023-11-02 09:01:56

Since OSidmap doesn’t have any key-value pair to pick linux-64 , there is no way to set it right now. There should be another parameter to switch between old and new formats here

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-02 09:09:17

Yeah not sure how to handle that. Any thoughts?

Erkan Erol
2023-11-02 09:29:50

I am trying to understand why the image doesn’t work for me when my vSphere is 7.0.3. Maybe I am missing something.

Erkan Erol
2023-11-02 09:30:31

To support both cases, we need to extend images/capi/hack/image-build-ova.py a little bit. @Yiyi Zhou What do you think?

Erkan Erol
2023-11-06 17:12:08

Hi again. I checked this issue in detail again. As far as I understand, the problem is flatcar specific. Flatcar should be mapped to other3xLinux64Guest instead of otherLinux64Guest

  • I tested with vSphere 7.0.3.01400 and it worked.

  • In the official Flatcar image, the value is other3xLinux64Guest See

  • I downloaded the latest stable OVA image of Flatcar and checked it too. It is other3xLinux64Guest there too.

  • The value in flatcar.json is already other3xLinux64Guest See


The weird part is the relationship between guestostype , vsphereguestostype , image-build-ova.py . It is really confusing. I didn’t get how it works.

image-builder-ova.py respects guest
ostype but there are some re-assignments in builder configuration like
"guest_os_type": "{{user vsphere&#95;guest&#95;os&#95;type}}",

for vsphere and vsphere-iso-base builders. See


image-builder-ova.py assumes the value of guest
ostype is a key of this map


OS_id_map = {"vmware-photon-64": {"id": "36", "version": "", "type": "vmwarePhoton64Guest"},
"centos7-64": {"id": "107", "version": "7", "type": "centos7_64Guest"},
"centos8-64": {"id": "107", "version": "8", "type": "centos8_64Guest"},
"rhel7-64": {"id": "80", "version": "7", "type": "rhel7_64Guest"},
"rhel8-64": {"id": "80", "version": "8", "type": "rhel8_64Guest"},
"rockylinux-64": {"id": "80", "version": "", "type": "rockylinux_64Guest"},
"ubuntu-64": {"id": "94", "version": "", "type": "ubuntu64Guest"},
"flatcar-64": {"id": "100", "version": "", "type": "otherLinux64Guest"},
"Windows2019Server-64": {"id": "112", "version": "", "type": "windows2019srv_64Guest"}}


Therefore, vsphere
guestostype should not be like other3xLinux64Guest and it shouldn’t overwrite guestostype but it does.

No idea how it does work.

Erkan Erol
2023-11-06 17:21:01

Since the value in the OVF file of recent images is otherLinux64Guest , I guess the script takes the right argument somehow. Therefore, I believe we need this change

Yiyi Zhou
2023-11-06 17:43:59

Thank you for getting to the bottom of this issue!

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-06 17:45:46

I think it’d be good to eventually refactor that script so that we don’t have those hardcoded values if possible. @Erkan Erol any chance you could open a basic issue for us to look at that in the future?

Erkan Erol
2023-11-08 08:07:51

I just opened an issue I have some urgent things for my daily job so I don’t have time for this issue now. I can check it later.

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-08 13:14:23

Just having the issue for now is a big improvement. Thanks for taking the time 😁

voor
2023-10-31 14:09:02

Cross posting since this seems related to image builder:

Robert Van Voorhees (https://kubernetes.slack.com/team/U2B8TBG58)
Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-31 15:05:47

There was a post a little way up about this, and a linked issue:

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)
Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-31 15:06:17

🙈 Which I see linked to your comment in the thread

Marcus Noble (k8s@marcusnoble.co.uk)
2023-10-31 15:09:38

@voor Would you mind updating with the latest outcome from your thread to include the things tried and ruled out? Would hate for it to get lost in the noise of Slack.

GitHub
Phil H
2023-11-03 16:31:22

@Phil H has joined the channel

libsysguy
2023-11-03 17:47:14

@libsysguy has left the channel

Jayesh
2023-11-06 08:32:58

Hi folks. Is there a provision by which I can build a RHEL kubernetes image for vsphere but instead of using an rhel iso as base, I can use a different format something like a vmdk?

Maarten
2023-11-06 10:06:35

@Maarten has joined the channel

Slackbot
2023-11-06 14:30:22

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles
2023-11-06 14:31:10

I'm away on holiday this week so won't make this.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-06 14:33:19

I’m at Kubecon and can’t make it

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-06 14:57:58

The agenda is empty and several people aren’t able to make it so let’s skip. 🙂

mboersma
2023-11-06 15:50:48

Kubecon week is always a tough time for community meetings, I agree let's skip until Nov. 20th.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-06 15:51:54

I’m currently sat in the contributor summit so I’ll try and convince more people to join us on image-builder! 😄

👍 mboersma, jsturtevant
😄 Erkan Erol
:megaman: jsturtevant
Jayesh
2023-11-07 04:58:20

Hi folks, Sorry for a follow up on this, any help would be appreciated.

Jayesh Srivastava (https://kubernetes.slack.com/team/U023B885W9M)
harrison_latimer
2023-11-08 14:48:48

@harrison_latimer has joined the channel

Jonas Pedersen
2023-11-12 15:19:21

@Jonas Pedersen has joined the channel

Jonas Pedersen
2023-11-12 15:24:28

Hello
Im trying to build and use an image for clusterApi to use to create k8s clusters with out direct internet connection. So Im using Nexus as a proxy. I have successfully created a image with version 1.28.3.
My challenge is that when I generate a cluster.yml for kubectl to apply the cluster that clusterApi creates still is trying to connect to registry.k8s.io. And that will not work in my setup. I cant seem to find where to change this kubeadm config yaml to use my nexus proxy.

Abhay Krishna Arunachalam
2023-11-12 17:10:49

Have you tried setting the field .spec.kubeadmConfigSpec.clusterConfiguration.imageRepository to your private registry?

Jonas Pedersen
2023-11-12 18:50:15

Yes I have configured the ansible template to point to my own repo. It is somehow sadly ignored or later overwritten

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-12 23:12:21

I’m currently travelling and on mobile so not got a link to hand but there was a very similar question a couple months ago in this channel that you might be able to find. Hopefully it contains the answer you need but I’m not 100% sure.

Abhay Krishna Arunachalam
2023-11-13 00:28:34

I think this is the thread Marcus is referring to.

https://kubernetes.slack.com/archives/C01E0Q35A8J/p1678141651934979

Jhonathan Cavalcante (https://kubernetes.slack.com/team/U02NR9XDWUT)
Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-16 11:17:48

I'm going to get a new release published so the latest fixes for vsphere are available to use. Speak up now if there's something you're also wanting to get included that isn't yet merged.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-16 11:31:22

Kicking off the process now. If there's some other changes we need we can always do another release 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-16 11:41:35

@mboersma I'm confused slightly about one step of the release process:


The image does end up in the staging repository but I don't see any new job appear in testgrid for it. 😕 Any idea why not?

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-16 11:45:06

Waiting on image promotion PRs being approved:

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-16 13:54:49

Docs PR:

GitHub
mboersma
2023-11-22 21:57:52

Hey @Marcus Noble, sorry I was out for a few days so didn't see this thread until now.

Yes, I don't see the jobs in testgrid either, although I've learned that the images will show up in staging anyway, I know the jobs did show up at one point after we nailed down the release process...I'm not sure what broke.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-22 21:59:27

The images are built as expected, I just cant find any logs / status showing them being built 🤷

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-16 13:48:01

Image-builder v0.1.21 is now available:
Thanks to all contributors! 💙 🎉

GitHub
:thank_you: mboersma
🎉 mboersma, nikparasyr
abdulraheem
2023-11-18 06:16:41

@abdulraheem has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-20 07:59:33

The image-builder office hours are due later today but there is currently only one item on the agenda (the announcement of the new release just above this post). If you have any topics you'd like to discuss please add them to the agenda. If it's still empty by the time the automated reminder message gets posted in here I'll conclude we have nothing to discuss this week and cancel it for today. 🙂

Slackbot
2023-11-20 14:30:23

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-20 14:38:33

☝️ Agenda still empty so I'm going to cancel for today! 🙂

Drew Hudson-Viles
2023-11-20 14:38:44

No problem 🙂

Martin
2023-11-21 21:36:58

@Martin has joined the channel

Martin
2023-11-21 21:37:59

I am trying to use a ubuntu image for vmware from here I don’t understand how the auth part works I don’t see any default passwords posted? I am I missing something ?

Dharmjit
2023-11-23 10:43:53

I guess the published node images don't have passwords, ssh keys and its expected to be used with cluster-api, infra-providers which set ups the required credentials for ssh access.

FG
2023-11-23 19:06:00

@FG has joined the channel

Slackbot
2023-11-25 17:05:02

This message was deleted.

Martin
2023-11-25 17:09:54

sorry about that posted to the wrong place, thanks

Marcus Noble (k8s@marcusnoble.co.uk)
2023-11-25 17:18:03

No worries 🙂

Slackbot
2023-12-04 14:30:13

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2023-12-04 15:16:14

☝️ The agenda is currently empty. Does anyone have anything they're like to discuss? If not I'm happy to skip but it's likely to be the last sync until next year.

mboersma
2023-12-04 15:45:55

I don't have any specific topics to discuss, but I can join if something gets added.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-12-04 16:22:01

Still no topics so I'm going to skip.

Happy holidays and hope you all enjoy the festive period! 🎉

👍 mboersma, jsturtevant
Drew Hudson-Viles
2023-12-04 15:17:05

I'm playing nurse to the family today so may not be able to join anyway I'm afraid! I've got some potential updates to nvidia bits incoming soon™ but that can wait until next year

:ack_tcp: Marcus Noble
👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2023-12-04 15:42:52

I've also realised that since the DST change that reminder in here is an hour earlier than it should be 🙈

Marcus Noble (k8s@marcusnoble.co.uk)
2023-12-04 15:43:47

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 2:30PM every other Monday (next occurrence is December 11th), Greenwich Mean Time.

:thank_you: mboersma, Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2023-12-04 15:44:19

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 3:30PM every other Monday (next occurrence is December 11th), Greenwich Mean Time.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-12-04 15:44:50

🤦‍♂️ I now don't know how to get it to start the reminder in two weeks

Marcus Noble (k8s@marcusnoble.co.uk)
2023-12-04 15:45:16

set up a reminder “from 18th December at 3:30pm Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 9AM every Monday, Greenwich Mean Time.

Marcus Noble (k8s@marcusnoble.co.uk)
2023-12-04 15:45:58

Nevermind, I'm going to set myself a reminder to create a new channel reminder in the new year 😆

😆 Drew Hudson-Viles
😄 mboersma
jsturtevant
2023-12-07 22:57:35

⚠️ I've opend a PR to remove the docker implementation related code for Windows. It was used with dockershim which was removed in 1.24 in k/k and relied on a bug in Docker Windows implementation for the support. This means containerd will be default (and only supported runtime for windows) in image builds. I believe most Windows implementations were already using containerd so should not be an issue

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2023-12-08 07:40:32

relied on a bug
🙈

:nod: jsturtevant
Shahar Shavit
2023-12-12 16:04:01

@Shahar Shavit has joined the channel

John Payne
2023-12-12 22:27:11

@John Payne has joined the channel

John Payne
2023-12-12 22:29:04

Hi all. If I am using a fresh ubuntu VHD, what else should I add to ensure that it can be used within an azure cluster?

jsturtevant
2023-12-13 00:44:05

are you using CAPZ?

jsturtevant
2023-12-13 00:46:44

Using image builder, if you run build-azure-vhd-ubuntu-2204 (or version equivalent ) it will prepare everything needed to run with CAPZ. If you are interested in the details you can browse the ansible that configures it

Shashinandan Srinivasa
2023-12-13 02:14:27

Hello Folks, v0.1.21 does not remove machine ID (/etc/machine-id) while building vsphere template(build-node-ova-vsphere-ubuntu-2204).. Anyone facing similar issue, or am I doing something wrong? This is causing all VMs being assigned same IP address. Anyone else facing same issue. I can see in build logs that it shows truncated, but it really isn't

Shashinandan Srinivasa
2023-12-13 02:14:43

vsphere-iso.vsphere: TASK [sysprep : Truncate machine id] *
2251 vsphere-iso.vsphere: changed: [default] => (item={'path': '/etc/machine-id', 'state': 'absent', 'mode': '0644'})
2252 vsphere-iso.vsphere: changed: [default] => (item={'path': '/etc/machine-id', 'state': 'touch', 'mode': '0644'})
2253 vsphere-iso.vsphere:
2254 vsphere-iso.vsphere: TASK [sysprep : Truncate hostname file] *

2255 vsphere-iso.vsphere: changed: [default] => (item={'path': '/etc/hostname', 'state': 'absent', 'mode': '0644'})
2256 vsphere-iso.vsphere: changed: [default] => (item={'path': '/etc/hostname', 'state': 'touch', 'mode': '0644'})

Shashinandan Srinivasa
2023-12-13 03:27:14

confirmed that building template using image-builder version v0.1.19 works fine.. v0.1.21 has issues

mboersma
2023-12-13 20:09:43

@Shashinandan Srinivasa could you open a GitHub issue for that?

Shashinandan Srinivasa
2023-12-14 17:39:04

Sure, running some more builds to see where it is going wrong. I will raise an issue after.

:thank_you: mboersma
Hugo Prudente
2023-12-13 15:28:16

@Hugo Prudente has joined the channel

Hugo Prudente
2023-12-13 15:30:24

Hi Folks,

I have opened a PR to support Centos9 Stream, We are heavy users of CentOS on Openstack and we are on a project with K8S it, for that I had to build the Centos 9 image to be compliant with our security and I'm returning it to the community.



Thanks.

👍 mboersma
Gong (Grace) Zhang
2023-12-15 07:02:07

@Gong (Grace) Zhang has joined the channel

linkarzu
2023-12-16 10:47:01

@linkarzu has joined the channel

mboersma
2023-12-18 15:24:30

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles
2023-12-18 15:25:22

I see no agenda... Enjoy the holidays?

mboersma
2023-12-18 15:25:36

If you have something image-builder to discuss, or any question to ask, please add it to the agenda above (or just mention it here in this Slack thread.)
If we don't have any topics, we'll skip until 2024. 🙂

mboersma
2023-12-18 15:26:20

Yes probably @Drew Hudson-Viles but I'm available if anyone has something to discuss.

Drew Hudson-Viles
2023-12-18 15:26:41

I suspect a number of people may have already broken up for the break.

I'm also available too. I appear to be one of the few people in my company still working this week 😄

mboersma
2023-12-18 15:27:03

Same here. 😀

Drew Hudson-Viles
2023-12-18 15:29:29
mboersma
2023-12-18 16:01:35

I think you're right @Drew Hudson-Viles; looks like we should skip this meeting.
But two Mondays from now is New Year's Day... Should we wait three weeks and start up again on January 8th?

Drew Hudson-Viles
2023-12-18 16:01:51

That sounds good to me buddy.

mboersma
2023-12-18 16:02:29

Let's do it. I hope you have excellent holidays, see you soon!

:party_wizard: Drew Hudson-Viles
Drew Hudson-Viles
2023-12-18 16:03:15

I was going to suggest the week after as it's unlikely anything major will crop up... but you always expect that over the holidays and it always happens 😛

So yes, let's stick with the 8th.

Have a good break my friend and I'll speak to you after.

🎄 mboersma
mboersma
2023-12-18 16:03:33

We don't have any agenda topics, so let's skip today's office hours. Because two weeks from now is New Year's Day, we'll have the next one on January 8th, 2024.
Happy Holidays everyone!

mboersma
2024-01-08 12:40:00

I think actually the 15th is when the next meeting is scheduled, not today. Sorry about the misinfo.

👍 Drew Hudson-Viles
Thomas Güttler
2023-12-25 18:30:13

@Thomas Güttler has joined the channel

nawazkh
2023-12-29 21:58:49

@nawazkh has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-08 10:01:11

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 3:30PM every other Monday (next occurrence is January 15th), Greenwich Mean Time.

:thank_you: mboersma
Danny Seymour
2024-01-08 21:01:23

@Danny Seymour has joined the channel

mcbenjemaa
2024-01-09 17:23:01

@mcbenjemaa has joined the channel

mcbenjemaa
2024-01-11 12:05:40

Hello image-builder maintainers,

Recently, I talked to SCL Leads regarding a new proposal for a new sub-project,

A Project for automating image builds using the Kubernetes API (Similar to CAPI but for images)

They mentioned that this is potentially an image-builder evolution.

Therefore, I'm sharing this with you for discussion and brainstorming.

Find my design proposal:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-11 12:52:30

Hey @mcbenjemaa 👋 This is great! I love seeing new ideas like this!

I have some thoughts / questions...

  • If I'm not mistaken, this would be more a replacement for Packer rather than specifically image-builder, yes? The "Why" in the document seems to conflate the two and I don't think that's quite correct. For example, image-builder provides end users with a known good (ideally) set of configurations for building a VM image for a kubernetes node. With how Forge is describe in the doc, it looks like it would be on the end users to come up with and configure the build steps themselves as it would be the CRs that they apply.

  • Following on from the previous - do you foresee a way to package and ship pre-build Build CRs that could then be used by end users so they don't need to know exactly everything that is needed for a Kubernetes node (e.g. replacing what image-builder does)

  • What is the benefit of having this as an operator? I don't really understand what value having a reconciliation loop provides in this case. I see the building of a VM image as a one-time thing - you configure it and build the artifact.

  • Do you have any thoughts on how variables might be used to allow users to configure things like package versions? I suspect this is what you're hinting at with mentions of Helm and Kustomize but it would be could to have that spelled out in the proposal.

  • I'd like to know more about the issues you state people have reported with using image-builder - Image-builder is a bit lazy in updating and upgrading dependencies. and Users also have asked different questions about the usage.. I'm not aware of specifically what these are referring too so if you have any sources etc. it would be great for us so that we can address them and improve things in this project.

  • Have you thought about how this might fit into a CI/CD pipeline? The current operator-based approach doesn't really lend itself well to that model. Specifically I'd like to see things like "Triggers" (I currently run image-builder automatically whenever there is a new GitHub release in kubernetes/kubernetes) , "Validation / Tests" (in image-builder we use goss to validate the state and at GS we're aiming to automatically run both our in-house E2E test suite and the Kubernetes conformance tests for every VM image we build) and "Post-build steps" (for example, replicating an AMI to multiple AZs and marking it as public or something like copying a built disk image to one or more S3 buckets).

  • Templating is not possible for packer templates nor ansible playbooks. - this isn't correct. Packer with HCL allows for templating.

  • Several of the CAPI provider teams run currently run image-builder nightly using Prow to test their applications / pre-test new releases. Have you thought about how Forge might handle that requirement? (I think this is a similar question to the CI/CD pipeline one)


Hopefully that's not too overwhelming 😆 I'm really excited to see how this goes!

Also... I'm shocked that there isn't already a CNCF / Kubernetes project called "Forge"! 😮

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-11 13:07:16

Also, we have the next image-builder office hours on Monday - do you think you'd be able to join and present this there too?

mcbenjemaa
2024-01-11 13:33:28

Yes, i will join.

👍 mboersma, Drew Hudson-Viles
🎉 Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-11 13:40:49

Awesome! 😄 i'll add it to the agenda then.

mcbenjemaa
2024-01-11 15:02:18

We are collecting thoughts and feedback.

The naming idea comes from a metaphor,
Forging Images
While the Core is an anvil, the infrastructure part is a blacksmith, and the provisioners are tools, like a Hammer.


First of All, as SCL leads stated, this could be an image-builder evolution like an image-builder v2.
However, Rebranding is a proposal whether it's approved or not.

Let me answer some of your questions:

If I'm not mistaken, this would be more a replacement for Packer rather than specifically image-builder, yes? The "Why" in the document seems to conflate the two and I don't think that's quite correct. For example, image-builder provides end users with a known good (ideally) set of configurations for building a VM image for a kubernetes node. With how Forge is describe in the doc, it looks like it would be on the end users to come up with and configure the build steps themselves as it would be the CRs that they apply.
This actually means a replacement for image-builder itself, and yes, this will get rid of Packer, but it still uses Ansible playbooks.
The aim here is to build the project and provide end-users with built-in templates so they can use them to build their images.
The proposal also mentions a CLI that can be used to help people migrate from packer/ansible to the Forge approach,
Which also means a migration plans for image-builder templates.

  • Following on from the previous - do you foresee a way to package and ship pre-build Build CRs that could then be used by end users so they don't need to know exactly everything that is needed for a Kubernetes node (e.g. replacing what image-builder does)

Yes, that's exactly what I was saying above. The image-builder will still have built-in templates or manifests. so people can use it.
With also a way of generating OR converting existing packer templates into BUILD CRs using the CLI.

  • What is the benefit of having this as an operator? I don't really understand what value having a reconciliation loop provides in this case. I see the building of a VM image as a one-time thing - you configure it and build the artifact.

this is summarized by the difference between the imperative and declarative approaches,
while the reconciliation loops are beneficial in terms of error detection and provide idempotent process,
However, there are also many out-of-the-box benefits of using CRDs, like Gitops, RBAC, Templating, etc.

Do you have any thoughts on how variables might be used to allow users to configure things like package versions? I suspect this is what you're hinting at with mentions of Helm and Kustomize but it would be could to have that spelled out in the proposal.
Well, many solutions could be used, like templating with kustomize or helm,
but also envsubst similar to CAPI cluster templates.

I'd like to know more about the issues you state people have reported with using image-builder - Image-builder is a bit lazy in updating and upgrading dependencies. and Users also have asked different questions about the usage.. I'm not aware of specifically what these are referring too so if you have any sources etc. it would be great for us so that we can address them and improve things in this project.

The word should be busy, upgrades are not very active.
The usage questions are teams internals with how to set up an automated pipeline to run image-builder and can build the same images for different regions, for example.

  • Have you thought about how this might fit into a CI/CD pipeline? The current operator-based approach doesn't really lend itself well to that model. Specifically I'd like to see things like "Triggers" (I currently run image-builder automatically whenever there is a new GitHub release in kubernetes/kubernetes) , "Validation / Tests" (in image-builder we use goss to validate the state and at GS we're aiming to automatically run both our in-house E2E test suite and the Kubernetes conformance tests for every VM image we build) and "Post-build steps" (for example, replicating an AMI to multiple AZs and marking it as public or something like copying a built disk image to one or more S3 buckets).


Actually, this is interesting.
I think the Build CRs should also be able to provide the image for multiple Regions.

Templating is not possible for packer templates nor ansible playbooks. - this isn't correct. Packer with HCL allows for templating.

This is actually should be re-phrase it,
Here I mean that we could not use features like a parent template, or bases such as with kustomize.

Several of the CAPI provider teams run currently run image-builder nightly using Prow to test their applications / pre-test new releases. Have you thought about how Forge might handle that requirement? (I think this is a similar question to the CI/CD pipeline one)

this is also an interesting point.
but can be solved by an e2e framework, which makes it possible to run using Prow.

jwest
2024-01-11 19:20:30

@jwest has joined the channel

Anurag
2024-01-12 14:40:26

@Anurag has joined the channel

Karine Santos
2024-01-12 17:45:08

@Karine Santos has joined the channel

Karine Santos
2024-01-12 17:48:29

hello everyone!
When trying to upload 2 images from k8s versions (1.26.12 and 1.28.5), I receive this error:

openstack: fatal: [default]: FAILED! => {"cache_update_time": 1705078887, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'kubelet=1.26.12-00' 'kubeadm=1.26.12-00' 'kubectl=1.26.12-00' 'kubernetes-cni=1.2.0-00'' failed: E: Version '1.26.12-00' for 'kubelet' was not found\nE: Version '1.26.12-00' for 'kubeadm' was not found\nE: Version '1.26.12-00' for 'kubectl' was not found\n", "rc": 100, "stderr": "E: Version '1.26.12-00' for 'kubelet' was not found\nE: Version '1.26.12-00' for 'kubeadm' was not found\nE: Version '1.26.12-00' for 'kubectl' was not found\n", "stderr_lines": ["E: Version '1.26.12-00' for 'kubelet' was not found", "E: Version '1.26.12-00' for 'kubeadm' was not found", "E: Version '1.26.12-00' for 'kubectl' was not found"], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nPackage kubeadm is not available, but is referred to by another package.\nThis may mean that the package is missing, has been obsoleted, or\nis only available from another source\n\nPackage kubelet is not available, but is referred to by another package.\nThis may mean that the package is missing, has been obsoleted, or\nis only available from another source\n\nPackage kubectl is not available, but is referred to by another package.\nThis may mean that the package is missing, has been obsoleted, or\nis only available from another source\n\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Package kubeadm is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", "", "Package kubelet is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", "", "Package kubectl is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", ""]}
5732 openstack:
5733 openstack: PLAY RECAP *
5734 openstack: default : ok=44 changed=34 unreachable=0 failed=1 skipped=190 rescued=0 ignored=0
5735 openstack:
5736==> openstack: Provisioning step had errors: Running the cleanup provisioner, if present...
5737==> openstack: Deleted temporary floating IP 'e89775ba-1edc-4d47-821a-bfc94582f064' (209.127.141.191)
5738==> openstack: Terminating the source server: 50ce0061-5572-4b00-b052-de0422138e0f ...
5739==> openstack: Deleting volume: d7d0b4c0-f238-4cc4-96f0-25fcd85bbb50 ...
5740==> openstack: Deleting temporary keypair: packer_65a16bce-18a3-3179-db2a-328434dae677 ...
5741Build 'openstack' errored after 19 minutes 48 seconds: Error executing Ansible: Non-zero exit status: exit status 2
5742==> Wait completed after 19 minutes 48 seconds
5743==> Some builds didn't complete successfully and had errors:
5744--> openstack: Error executing Ansible: Non-zero exit status: exit status 2
5745==> Builds finished but no artifacts were created.
5746make[1]:
* [Makefile:500: build-openstack-ubuntu-2204] Error 1
5747make[1]: Leaving directory '/builds/magalu-cloud-iaas/k8s/capi-image-builder'
5748+ clean_up_images
5749+ echo 'Starting image clean up'
5750Starting image clean up
5751+ grep -q -v -e '^$' ./images_list.tmp
5752+ echo 'No images to delete'
5753No images to delete
Do you know what it could be?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-12 19:07:14

I’m currently not near my laptop so can’t double check but I suspect it might be related to the Kubernetes package repo change.
What version of image-builder are you using and what does your vars look like?

Drew Hudson-Viles
2024-01-12 19:10:46

I was about to suggest the same. I think it is related to the repos and that 1.26 isn't available in the current ones. Can you try a build with 1.27 just to check? I suspect that will work.

Karine Santos
2024-01-12 20:30:37

Hello @Marcus Noble and @Drew Hudson-Viles !!!
Today we use a clone of image-builder and I believe it is not that up to date.
Can you help me where the version of the image builder and vars are defined in this repo?

GitHub
Drew Hudson-Viles
2024-01-12 20:37:39

Hi,

So the releases page will show you the latest versions available however one quick check you can do is to check the value of the repo you have defined in your fork - see this PR for the change that went in to update this and the file in which you can check.


This was updated back in September to reflect the new repos as the old ones were frozen and were expected to be removed in January of this year

Karine Santos
2024-01-15 09:38:00

Hello @Marcus Noble and @Drew Hudson-Viles !!!

@Drew Hudson-Viles For version 1.27 I get the error:

ERROR: python-cinderclient 9.4.0 has requirement requests>=2.25.1, but you'll have requests 2.22.0 which is incompatible.
3682Installing collected packages: pbr, stevedore, wcwidth, PrettyTable, zipp, importlib-metadata, autopage, pyperclip, attrs, cmd2, cliff, iso8601, oslo.i18n, netifaces, packaging, pytz, tzdata, pyparsing, wrapt, debtcollector, oslo.utils, msgpack, oslo.serialization, os-service-types, keystoneauth1, python-novaclient, rfc3986, oslo.config, python-keystoneclient, requestsexceptions, platformdirs, decorator, typing-extensions, dogpile.cache, jsonpointer, jsonpatch, openstacksdk, osc-lib, python-cinderclient, python-openstackclient
3683Successfully installed PrettyTable-3.9.0 attrs-23.2.0 autopage-0.5.2 cliff-4.5.0 cmd2-2.4.3 debtcollector-2.5.0 decorator-5.1.1 dogpile.cache-1.3.0 importlib-metadata-7.0.1 iso8601-2.1.0 jsonpatch-1.33 jsonpointer-2.4 keystoneauth1-5.5.0 msgpack-1.0.7 netifaces-0.11.0 openstacksdk-2.1.0 os-service-types-1.7.0 osc-lib-3.0.0 oslo.config-9.3.0 oslo.i18n-6.2.0 oslo.serialization-5.3.0 oslo.utils-6.3.0 packaging-23.2 pbr-6.0.0 platformdirs-4.1.0 pyparsing-3.1.1 pyperclip-1.8.2 python-cinderclient-9.4.0 python-keystoneclient-5.3.0 python-novaclient-18.4.0 python-openstackclient-6.4.0 pytz-2023.3.post1 requestsexceptions-1.4.0 rfc3986-2.0.0 stevedore-5.1.0 typing-extensions-4.9.0 tzdata-2023.4 wcwidth-0.2.13 wrapt-1.16.0 zipp-3.17.0

@Marcus Noble for my repo:
cni.json:
{
"kubernetes_cni_deb_version": "1.2.0-00",
"kubernetes_cni_http_checksum": "sha256: kubernetes&#95;cni&#95;http&#95;checksum&#95;arch}}-v1.2.0.tgz.sha256",
"kubernetes_cni_http_checksum_arch": "amd64",
"kubernetes_cni_http_source": "",
"kubernetes_cni_rpm_version": "1.2.0-0",
"kubernetes_cni_semver": "v1.2.0",
"kubernetes_cni_source_type": "pkg"
}

packerkubernetes.json :
{
"crictl_arch": "amd64",
"crictl_sha256": " crictl&#95;version}}/crictl-v{{user crictl&#95;version}}-linux-{{user crictl&#95;arch}}.tar.gz.sha256",
"crictl_source_type": "pkg",
"crictl_url": " crictl&#95;version}}/crictl-v{{user crictl&#95;version}}-linux-{{user crictl&#95;arch}}.tar.gz",
"crictl_version": "{{env CRICTL&#95;VERSION}}",
"kubeadm_template": "etc/kubeadm.yml",
"kubernetes_container_registry": "registry.k8s.io",
"kubernetes_deb_gpg_key": "",
"kubernetes_deb_repo": "\" kubernetes-xenial\"",
"kubernetes_deb_version": "{{env KUBE&#95;VERSION}}-00",
"kubernetes_http_source": "",
"kubernetes_load_additional_imgs": "false",
"kubernetes_rpm_gpg_check": "True",
"kubernetes_rpm_gpg_key": "\" \"",
"kubernetes_rpm_repo": " kubernetes&#95;rpm&#95;repo&#95;arch}}",
"kubernetes_rpm_repo_arch": "x86_64",
"kubernetes_rpm_version": "{{env KUBE&#95;VERSION}}-0",
"kubernetes_semver": "v{{env KUBE&#95;VERSION}}",
"kubernetes_series": "v{{env KUBE&#95;SERIES}}",
"kubernetes_source_type": "pkg",
"systemd_prefix": "/usr/lib/systemd",
"sysusr_prefix": "/usr",
"sysusrlocal_prefix": "/usr/local"
}
for my variables:
CRICTL
VERSION: "1.26.1"
KUBEVERSION : "1.26.12" and "1.28.5"
KUBE
SERIES : "1.26" and "1.28"

Drew Hudson-Viles
2024-01-15 09:43:50

So it does look like being out of date is an issue here.

for 1.27 I can see you're getting ERROR: python-cinderclient 9.4.0 has requirement requests>=2.25.1, but you'll have requests 2.22.0 which is incompatible.

For the other info you've provided you are using

 "kubernetes_deb_repo": "\" kubernetes-xenial\"",
Where as it should now be
"kubernetes_deb_repo": " user kubernetes&#95;series }}/deb/",

I think as it's a fork you're options here are to manually update the fields or sync the fork with the main image-builder repo as it appears it is out of date.

Unless Marcus can suggest anything else of course! 😉

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-15 09:47:32

That sounds sensible to me.

Karine Santos
2024-01-15 12:26:55

I changed the:

"kubernetes_deb_repo": " user kubernetes&#95;series }}/deb/",
and also the ansible_args.json:
openstack: fatal: [default]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/home/ubuntu/~core/.ansible/tmp/ansible-tmp-1705320189.7624567-59973788767630/AnsiballZ_apt_repository.py\", line 102, in \n    _ansiballz_main()\n  File \"/home/ubuntu/~core/.ansible/tmp/ansible-tmp-1705320189.7624567-59973788767630/AnsiballZ_apt_repository.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/ubuntu/~core/.ansible/tmp/ansible-tmp-1705320189.7624567-59973788767630/AnsiballZ_apt_repository.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.packaging.os.apt_repository', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib/python3.10/runpy.py\", line 224, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.10/runpy.py\", line 96, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_apt_repository_payload_7ka7dvjo/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 564, in \n  File \"/tmp/ansible_apt_repository_payload_7ka7dvjo/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 547, in main\n  File \"/usr/lib/python3/dist-packages/apt/cache.py\", line 152, in __init__\n    self.open(progress)\n  File \"/usr/lib/python3/dist-packages/apt/cache.py\", line 214, in open\n    self._cache = apt_pkg.Cache(progress)\napt_pkg.Error: E:Malformed entry 1 in list file /etc/apt/sources.list.d/kubernetes.list (Component), E:The list of sources could not be read.\nConnection to 127.0.0.1 closed.\r\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Karine Santos
2024-01-15 14:19:49

Any suggestions as to what it could be @Marcus Noble @Drew Hudson-Viles?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-15 14:22:05

Connection to 127.0.0.1 closed. That sounds like its possibly network error. Does this consistently fail the same if you re-run it?

Drew Hudson-Viles
2024-01-15 14:25:09

To add to that you also have E:Malformed entry 1 in list file /etc/apt/sources.list.d/kubernetes.list (Component), E:The list of sources could not be read and so I suspect many values have been changed since you've last synced with the project and that changing those few lines would be enough.

I'd recommend bringing the latest changes into your project via a sync with the upstream and then if you have any custom requirements on top of that, ensure they are still valid and work with where the project is at now.

If you currently have changes in your main branch, the easiest approach would be to check them out into another branch so that you don't lose anything, sync with upstream and then PR those changes back in.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-15 14:26:57

Alternatively, if you don't have any code changes in your fork and only changes to vars I'd recommend using the container image that we build for each new release. Then you can be sure that the versions of binaries used are ok too.

👍 Drew Hudson-Viles
Karine Santos
2024-01-16 13:55:22

Hello @Drew Hudson-Viles and @Marcus Noble!!!
Thank you for your help! I followed exactly these updates here and managed to upload my new images.

GitHub
🙌 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-16 14:58:57

🎉 That mean you were able to successfully build the images? 😄

mcbenjemaa
2024-01-15 15:07:23

I'm trying to build QEMU images with image-builder from a GitHub action job,
But I got this error:

The builder qemu is unknown by Packer, and is likely part of a plugin that is
not installed.
You may find the needed plugin along with installation instructions documented
on the Packer integrations page.



Error: Failed to initialize build "qemu"

Shouldn;t the packer plugin for QEMU is part of deps-qemu ?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-15 15:12:25

I'm guessing you're on a newer version of Packer than is supported by image-builder as the qemu plugin was built-in until v1.10.0.
Are you using the provided image-builder container image? If so I suspect we have something wrong in the deps scripts.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-15 15:14:05

v1.9.5 of Packer is the latest supported by image-builder due to the licence change.

👍:skin_tone_3: mcbenjemaa
mcbenjemaa
2024-01-15 15:23:33

yeah, Packer 1.10.0

mcbenjemaa
2024-01-15 15:24:43

I can use the container image?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-15 15:25:17

You should be able to. I've never done it in a GitHub action but I don't see why it wouldn't work.

mcbenjemaa
2024-01-15 15:27:42

, yeah i will try it. Thanks

mcbenjemaa
2024-01-15 15:28:06

btw, is the office hours 16:30 Or 17:30?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-15 15:29:13

16:30 UTC

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-15 15:29:45

There should be a reminder message posted in 1 minute 😛

😁 mcbenjemaa
mcbenjemaa
2024-01-15 15:32:59

yeah, starts in 1 hour.

mcbenjemaa
2024-01-15 15:34:45

docker run --name image-builder <br> registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.21 make build-qemu-ubuntu-2204
is the container run, something like that?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-15 15:35:51

Yeah. You'll need to mount in the needed vars files and maybe some env vars, depending on how you're configuring, but that's pretty much it

mcbenjemaa
2024-01-15 15:36:32

right.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-19 07:31:43

We've had an issue opened about this version problem:
I've created a new issue to update image-builder to restrict the versions of Packer is will run with:

mcbenjemaa
2024-01-19 13:34:35

Good initiative

mcbenjemaa
2024-01-19 13:38:25

I have an update regarding the Forge project proposal:

The SCL leads are not very keen, that both projects coexists.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-19 13:40:23

What was their reasoning?

mcbenjemaa
2024-01-19 14:39:38
The fact that people need time to migrate is acceptable, but in order to make it happen there should be some sort of strong intent leading to feature parity in a reasonable time and to a plan with a sustainable deadline for the phase off, and unfortunately it seems to me we are missing both (intent and feature parity expectation)
I know that gaining consensus takes time, but now this smells like “yeah, nice call, but ultimately they will co-exist indefinitely” which is something I’m not really happy about
mcbenjemaa
2024-01-19 14:40:44

I'm looking now for consensus within the community.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-19 14:45:30

I know that gaining consensus takes time, but now this smells like “yeah, nice call, but ultimately they will co-exist indefinitely” which is something I’m not really happy about
I disagree with this. I think having options that suit different needs is totally fine. Just look at all the different ways of creating clusters and the different ways of running Kubernetes. There is no one solution for everyone. 🤷

mboersma
2024-01-19 20:18:10

Yes, I thought we had the idea of experimental or incubator projects to accommodate things like that.
I agree overall that if we want to replace image-builder someday forge has to aim for feature parity, which seems like a lot to do. But I don't think there would be a huge amount of resistance to deprecating image-builder in favor of forge if it met the same requirements.

👍:skin_tone_3: mcbenjemaa
Slackbot
2024-01-15 15:30:30

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-15 15:40:55

☝️ In just under an hour we'll have @mcbenjemaa giving an introduction to Forge that he shared a few days ago. Everyone is welcome to join. If anyone has any other discussion topics please add them to the agenda or let me know. 🙂 See y'all soon.

Mohamed chiheb ben jemaa (https://kubernetes.slack.com/team/UF111SQ4U)
👍 Drew Hudson-Viles, mboersma
:fire_blue: mcbenjemaa
mboersma
2024-01-15 18:12:05

A recording of today's image-builder office hours is available on YouTube:

mboersma
2024-01-16 14:48:29

Image-builder v0.1.22 is now available:
Thanks to all contributors! 💙 🎉

🙌 Karine Santos
:parrotk8s: Marcus Noble, mloskot, Drew Hudson-Viles, jsturtevant, Magnus RC, Anurag
🙌:skin_tone_2: Anurag
:_thank_you_: Anurag
:blod_tada: Anurag
Abhay Krishna Arunachalam
2024-01-19 07:35:44

This is an issue opened a month ago, and now we're facing it too. Could someone take a look? I have added a comment with some of my findings.

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-19 07:49:07

If I understand your comment it looks like you've found the solution, yes? Would you be willing to open a PR with the change to using nmcli?

Abhay Krishna Arunachalam
2024-01-19 08:35:35

Thanks for your time reading my comment. Here you go

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-19 08:36:01

Wow that was quicker than I expected 😆 Thanks!

Abhay Krishna Arunachalam
2024-01-19 08:36:17

I kept the PR description terse as I have referenced my comment, hope that's okay

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-19 08:37:07

Is the Reset network interface IDs task still needed after this change?

Abhay Krishna Arunachalam
2024-01-19 08:43:42

I believe so because this change doesn't affect the behavior of that task, it just puts the nmconnection file in place so that the sed command can find it and delete the uuid.

Here are the contents of the file before the change, the Reset network interface IDs removes the uuid

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-19 08:44:19

Gotcha! So it makes sure that dir is setup and then the next task is able to do its thing 🙂 👍

Abhay Krishna Arunachalam
2024-01-19 08:45:17

exactly, I saw that even without the nmcli change, the /etc/NetworkManager/system-connections dir is present but it's empty as the machine was still using the ifcfg files

Abhay Krishna Arunachalam
2024-01-19 08:45:35

Thanks for the lgtm! 🚀

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-19 08:45:45

I've added my lgtm. I'll leave it for one of the other maintainers to add theirs too once the tests pass if thats cool with you 🙂

Abhay Krishna Arunachalam
2024-01-19 08:47:17

Yeah that sounds good!

👍 Marcus Noble
Yongxiang Gao
2024-01-22 22:41:29

@Yongxiang Gao has joined the channel

Yongxiang Gao
2024-01-22 22:44:45

On an Azure ubuntu VM node, I try to build ubuntu image 2004 for bare metal (raw) with full disk encryption support, however, the build stuck here:

==> qemu: Retrieving ISO
==> qemu: Trying
==> qemu: Trying
==> qemu: => /root/.cache/packer/48e4ec4daa32571605576c5566f486133ecc271f.iso
==> qemu: Starting HTTP server on port 8529
==> qemu: Found port for communicator (SSH, WinRM, etc): 3319.
==> qemu: Looking for available port between 5900 and 6000 on 127.0.0.1
==> qemu: Starting VM, booting from CD-ROM
qemu: The VM will be run headless, without a GUI. If you want to
qemu: view the screen of the VM, connect via VNC without a password to
qemu:
==> qemu: Waiting 10s for boot...
==> qemu: Connecting to VM via VNC (127.0.0.1:5986)
==> qemu: Typing the boot commands over VNC...
qemu: Not using a NetBridge -- skipping StepWaitGuestAddress
==> qemu: Using SSH communicator to connect: 127.0.0.1
==> qemu: Waiting for SSH to become available...

How to trouble shoot, is there something like remote/web console in vcode of vSphere build to check the VM status?

Yongxiang Gao
2024-01-23 02:41:30

Look like there are following kvm process running:

root   3807270 3807149 3 00:17 pts/0  00:02:03 /usr/bin/qemu-system-x86_64 -smp 1 -drive if=none,file=output/ubuntu-2004-kube-v1.26.7/ubuntu-2004-kube-v1.26.7,id=drive0,cache=writeback,discard=unmap,format=raw -drive file=/root/.cache/packer/48e4ec4daa32571605576c5566f486133ecc271f.iso,media=cdrom -boot once=d -vnc 127.0.0.1:15 -m 2048M -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive0 -device virtio-net,netdev=user.0 -machine type=pc,accel=kvm -bios OVMF.fd -name ubuntu-2004-kube-v1.26.7 -netdev user,id=user.0,hostfwd=tcp::3837_:22
root 3807273 2 0 00:17 ? 00:00:00 [kvm-nx-lpage-re]
root 3807276 2 0 00:17 ? 00:00:00 [kvm-pit/3807270]

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-23 06:45:34

I haven't seen this issue before but also not seen image-builder run in Azure -> vSphere myself so not 100% sure if you've hit a known bug or not.

Just to confirm, are you using the latest release of image-builder?

The Waiting for SSH to become available... makes me suspect it might be some form of firewall blocking access. Are you able to confirm that port 22 should be open between Azure and vSphere?

Yongxiang Gao
2024-01-23 09:25:32

Yes, I use the latest image-builder source code plus the patches from eks-anywhere-build-tooling for image-builder.

The issue maybe related to encryption change I put in preseed-efi.cfg, the process chain is like this (parent down to child):

  1. make build-raw-ubuntu-2004-efi

  2. packer build ...

  3. packer build ...

  4. packer plugin packer-builder-qemu

  5. qemu-system-x8664 -smp 1 -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive0 -device virtio-net,netdev=user.0 -name ubuntu-2004-kube-v1.26.7 -machine type=pc,accel=kvm -vnc 127.0.0.1:89 -m 2048M -netdev user,id=user.0,hostfwd=tcp::3223:22 -bios OVMF.fd -drive if=none,file=output/ubuntu-2004-kube-v1.26.7/ubuntu-2004-kube-v1.26.7,id=drive0,cache=writeback,discard=unmap,format=raw -drive file=/root/.cache/packer/48e4ec4daa32571605576c5566f486133ecc271f.iso,media=cdrom -boot once=d

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-23 09:28:12

patches from eks-anywhere-build-tooling for image-builder.
What are these? I've never come across these before 😮

Based on what you say it does sound like the encryption might be related. Are you able to test the same config without the encryption to see if that succeeds?

Yongxiang Gao
2024-01-23 09:47:33

The fix some issues and some improvements, make it EKSA conformant.

Yongxiang Gao
2024-01-23 09:48:13

Right, without the encryption related changes, the build pass

Yongxiang Gao
2024-01-23 09:49:02

Do you know someone(s) have working solution to support full-disk encryption?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-23 09:49:47

Not that I've seen. Doesn't look like there's any related issues either.

Yongxiang Gao
2024-01-23 09:52:31

I know we can switch to full disk encryption when install ubuntu on individual machine manually.
However, I would like to do it automatically to create EKS anywhere cluster on bare metal.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-23 09:54:55

Yeah I get that. Not sure what is currently blocking it. I would think that Packer would be possible with full disk encryption but perhaps there's some configuration or flag that we're currently missing.

Yongxiang Gao
2024-01-23 17:43:23

How can I get familiar with packer and qemu-system-x86_64 quickly?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-23 17:45:42

For Packer the docs cover quite a lot of stuff if you search for things -
For qemu - I have no idea. I know next to nothing about it myself. 😞

:ty: Yongxiang Gao
Drew Hudson-Viles
2024-01-31 16:11:28

I've just tried this locally as I'm testing a build for KubeVirt and it looks like on my side it's getting stuck at the Select Language screen. I'll see if I can spot why.

Drew Hudson-Viles
2024-01-31 16:12:13

Also ,this is just QMEU w/22.04-efi but I suspect something similar is happening as it's hanging at the Waiting for SSH to become available prompt.

Drew Hudson-Viles
2024-01-31 16:25:11

ok so in my case I thought I'd fat fingered a config but it seems on reboot it's not ejecting the "cdrom" and so it's booting back into the installer. This prevents the next phase from running.

Lemmie see if I can solve this.

Drew Hudson-Viles
2024-01-31 17:14:25

This is funky - it seems it's working now and I've all I've done so far is increased the disk size double_facepalm . The efibootorder is obvioulsy doing it's job but I wonder if disk space was an issue with the default of 2G... Not certain tbh but it's all I've changed.

Maybe I'm not cut out for this computer stuff 😄

👍 Yongxiang Gao
Yongxiang Gao
2024-02-01 17:50:49

Please share your changes in detail. ty

Drew Hudson-Viles
2024-02-01 18:06:36

Hi, I've sent you a DM with the details but in case anyone else comes across this, my changes are in this PR.

https://github.com/kubernetes-sigs/image-builder/pull/1389

GitHub
Jasper
2024-01-24 09:24:09

@Jasper has joined the channel

Yongxiang Gao
2024-01-25 18:31:51

Where (the source code) are these messages from?
# make build-raw-ubuntu-2004-efi
....

==> qemu: Retrieving ISO
==> qemu: Trying
==> qemu: Trying

Abhay Krishna Arunachalam
2024-01-25 18:37:51

👍 Yongxiang Gao
Yongxiang Gao
2024-01-25 22:42:29

@Abhay Krishna Arunachalam I try to support full disk encryption with change in preseed-efi.cfg as follows:

diff --git a/images/capi/packer/raw/linux/ubuntu/http/base/preseed-efi.cfg b/images/capi/packer/raw/linux/ubuntu/http/base/preseed-efi.cfg
index 14cb4008f..fca87df75 100644
--- a/images/capi/packer/raw/linux/ubuntu/http/base/preseed-efi.cfg
+++ b/images/capi/packer/raw/linux/ubuntu/http/base/preseed-efi.cfg
@@ -52,7 +52,12 @@ d-i partman-partitioning/default_label string gpt
d-i partman/choose_label string gpt
d-i partman/default_label string gpt

-d-i partman-auto/method string regular
+#d-i partman-auto/method string regular
+d-i partman-auto/method string crypto
+d-i partman-crypto/confirm boolean true
+d-i partman-crypto/method string luks
+d-i partman-crypto/passphrase password possible
+d-i partman-crypto/passphrase-again password possible
d-i partman-auto/choose_recipe select gpt-boot-root-swap
d-i partman-auto/expert_recipe string <br> gpt-boot-root-swap :: <br>@@ -78,6 +83,8 @@ d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

+d-i initramfs-tools/cryptroot-initramfs-tools/verbose boolean true
+
# Create the default user.
d-i passwd/user-fullname string builder
d-i passwd/username string builder
@@ -93,6 +100,9 @@ d-i grub-installer/with_other_os boolean true
d-i finish-install/reboot_in_progress note
d-i pkgsel/update-policy select none

+d-i debian-installer/add-kernel-opts string <br>+ "cryptopts=target=root,source=/dev/sda3,luks"

The build stuck at waiting for SSH:
==> qemu: Connecting to VM via VNC (127.0.0.1:5975)
==> qemu: Typing the boot commands over VNC...
qemu: Not using a NetBridge -- skipping StepWaitGuestAddress
==> qemu: Using SSH communicator to connect: 127.0.0.1
==> qemu: Waiting for SSH to become available...

Am I missing something?

Abhay Krishna Arunachalam
2024-01-25 22:43:28

Did you not face this error before these changes?

Yongxiang Gao
2024-01-25 23:48:25

Right, the build passes if I don't put the changes there

Yongxiang Gao
2024-01-27 08:37:23

Something wrong with packer-plugin-qemu builder, especially at stepTypeBootCommand

Michi Altstaedt
2024-01-26 09:38:26

@Michi Altstaedt has joined the channel

Michi Altstaedt
2024-01-26 09:55:19

[k8s 1.29.1] which versions i must put in json for kubernetescnideb_version? I fail to determine what is available and what is compatible to this k8s version.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-26 10:36:20

I think you will want something like this:

{
"kubernetes_deb_version": "1.29.1-00",
"kubernetes_rpm_version": "1.29.1-0",
"kubernetes_semver": "v1.29.1",
"kubernetes_series": "v1.29",
}

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-26 10:36:39

Those were what I used to built from Flatcar on AWS.

Anurag
2024-01-26 11:10:22

If you're using pkgs.k8s.io then this might be helpful to you.


In general, I was not able to find a page where it's written which CNI version to use with an upstream kubernetes release.
Matt suggested to use a container and find out and I think you'll have to do the same. I like/use this same approach.

Having said that, with 1.29 series, you'll use 1.3.0 one as kubernetescnideb_version

Michi Altstaedt
2024-01-26 11:15:21

thanks @Anurag, that worked for me as well. I used these values:

    "kubernetes_cni_deb_version": "1.3.0-1.1",
"kubernetes_cni_http_checksum": "sha256: kubernetes&#95;cni&#95;http&#95;checksum&#95;arch}}-v1.3.0.tgz.sha256",
"kubernetes_cni_http_checksum_arch": "amd64",
"kubernetes_cni_http_source": "",
"kubernetes_cni_rpm_version": "1.3.0",
"kubernetes_cni_semver": "v1.3.0",
"kubernetes_cni_source_type": "pkg",

:blod_tada: Anurag
mboersma
2024-01-29 15:51:32

I'm trying to work out a more general solution to this (see bug #1363) but no luck yet. Ideas are appreciated!
The only change I know I'd like to make is to stop specifying the kubernetescnideb_version and just allow apt to figure out which version goes with the version of Kubernetes we're currently installing. (The user could still override that to choose a specific CNI package.) But I haven't got something similar working for rpms, and it doesn't help the "install from source" path that we also support for k8s+cni.

Slackbot
2024-01-29 15:30:16

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-29 15:31:45

☝️ The agenda is currently empty. Does anyone have anything they've like to raise? If not I'm in favour of skipping this week.

mcbenjemaa
2024-01-29 15:41:39

I just want to say that I will start a prototype for the Forge project with 2 volunteers before proposing to SCL.

And if anyone is interested in getting involved in the prototype, he is welcome to join.

nothing more 😄

👍 mboersma
mboersma
2024-01-29 15:46:12

I don't have anyhing in particular to discuss, but I'm happy to be there if anyone does. I added a note about the v0.1.22 release.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-01-29 15:51:11

If there's nothing else I think I'm going to give it a miss. I have another meeting after it and could do with a break today

👍 mboersma
👍:skin_tone_3: mcbenjemaa
Yongxiang Gao
2024-02-01 17:56:05

How to separate certain directory like /var into another partition and make this partition encrypted?
I try to add new partition for /var in file images/capi/packer/raw/linux/ubuntu/http/base/preseed-efi.cfg

However, the output image file doesn't have the new partition.

Actually, in packer-plugin-qemu, it seems that, with ubuntu-2004 and ubuntu-2004-efi, the output image file is created from the same ISO image, but, the the output image file are really different (w/wo efi), how are the preseed files or other changes applied to it at all?

Drew Hudson-Viles
2024-02-02 08:34:45

Hi

It's passed in via userdata:


You should only need to make changes to the base/preseed-efi.cfg to make the required changes you desire.

The pressed is then passed in via the boot commands

Yongxiang Gao
2024-02-02 10:10:40

Thanks.
However, actually include
d-i preseed/include string ../base/preseed-efi.cfg

And I tried to add partitions like this in my local file packer/raw/linux/ubuntu/http/base/preseed-efi.cfg:

d-i partman-auto/expert_recipe string     <br>  gpt-boot-root ::              <br>   1 1 1 free               <br>     $bios_boot{ }            <br>     method{ biosgrub } .         <br>   200 200 200 fat32            <br>     $primary{ }             <br>     method{ efi } format{ } .      <br>   # 512 512 512 ext3            <br>   #  $primary{ } $bootable{ }       <br>   #  method{ format } format{ }      <br>   #  use_filesystem{ } filesystem{ ext3 } <br>   #  mountpoint{ /boot } .        <br>   5120 20000 -1 ext4           <br>     $primary{ } $bootable{ }      <br>     method{ format } format{ }      <br>     use_filesystem{ } filesystem{ ext4 } <br>     mountpoint{ / } .          <br>   1024 2048 4096 ext4         <br>     method{ format } format{ }      <br>     use_filesystem{ } filesystem{ ext4 } <br>     mountpoint{ /home } .        <br>   1024 2048 4096 ext4         <br>     method{ format } format{ }      <br>     use_filesystem{ } filesystem{ ext4 } <br>     mountpoint{ /var } . 

The changes has no effect in the output image file: output/ubuntu-2004-kube-v1.26.7/ubuntu-2004-kube-v1.26.7, which always has 2 partitions:

Device                            Start    End  Sectors  Size Type
output/ubuntu-2004-kube-v1.26.7/ubuntu-2004-kube-v1.26.7p1 34 1050815 1050782 513.1M EFI System
output/ubuntu-2004-kube-v1.26.7/ubuntu-2004-kube-v1.26.7p2 1050816 16678878 15628063 5.5G Linux filesystem

Drew Hudson-Viles
2024-02-02 10:40:21

From what I can see, the preseed you have looks fine. I can't see any reason this would not working at a quick glance and I don't have time to test this locally at the moment but will when I get chance.

Can you connect via VNC (if that's an option during RAW builds) and see what's happening on each step? Can you see that step running?

Can you also confirm you've changed the line here to match the name of your recipe?

Yongxiang Gao
2024-02-02 11:33:09

Good catch.
However, after I change the name back to the original, the new partitions are still missing in the output image file.

Yongxiang Gao
2024-02-02 11:41:54

I will resort to VNC for sure.

👍 Drew Hudson-Viles
Pengfei Huang
2024-02-03 23:44:41

@Pengfei Huang has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-07 10:27:19

FYI: I'm creating a new image-builder release that includes this change to address a CVE 🙂 -

GitHub
👍 Drew Hudson-Viles, mboersma
👍:skin_tone_3: mcbenjemaa
Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-07 10:37:31

@Drew Hudson-Viles Do you want to see if you have the permissions to be able to /approve this PR? (I'm not sure if it's just maintainers or if reviewers are also able to)

GitHub
Drew Hudson-Viles
2024-02-07 11:23:56

sure one tick

Drew Hudson-Viles
2024-02-07 11:24:22

oh Matt beat me to it 😄

😆 Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-07 11:30:04

Documentation update -

GitHub
👍 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-07 11:34:16

Can I get an /approve too please? 🙂

Drew Hudson-Viles
2024-02-07 11:34:24

I can try 😛

Drew Hudson-Viles
2024-02-07 11:34:51

DENIED 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-07 11:34:54

😭

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-07 11:35:43

I'm sure @mboersma will get to it when he's about 🙂 I'm about to head AFK but once that PR is merged could someone please announce it in the main channel? 🙏 The release is already created -

GitHub
👍 Drew Hudson-Viles
mboersma
2024-02-07 15:03:33

Thanks for pouncing on this @Marcus Noble and @Drew Hudson-Viles!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-07 17:49:57

I was just happy to see someone else had already done the PR with the CVE fix when I got there! 😉

Drew Hudson-Viles
2024-02-07 12:49:05

Image-builder v0.1.23 is now available:
Thanks to all contributors!

:thank_you: mboersma, Anurag
🎉 mboersma, Yongxiang Gao, Travis Holton
:partyk8s: mboersma
Yongxiang Gao
2024-02-08 20:27:29

Does image-builder support full disk encryption?

Drew Hudson-Viles
2024-02-08 20:32:22

It's not really image builder that would provide this but if it can be done via the preseed/cloud-init then yes it should support it.

Yongxiang Gao
2024-02-08 21:52:29

In case we are using the image (from image-builder) in tinkerbell action to create a EKS anywhere cluster on bare metal hosts.
What shall we do if we want to support encryption?

Yongxiang Gao
2024-02-08 22:01:04

The simplest way is to use the image (with preseed.cfg taken inside already), because we usually use stream-image action to install the OS.

Yongxiang Gao
2024-02-08 22:07:44

The problem is that, if I add the encryption part in the preseed-efi.cfg, I will get image-builder stuck as I reported on Jan 22 2024
I just file a ticket for such issue:

GitHub
Drew Hudson-Viles
2024-02-08 22:46:04

Honestly, I'm not sure about this as I've not used preseed to setup encryption before.
When I get time I can take a look into it unless someone else comes up with something to help before then.
However, if it's hanging at waiting for SSH then it suggests an issue with the preseed itself which prevents it completing successfully. This means the VM will never reboot and launch using the generated disk image so that image-builder can proceed with the installation.

Yongxiang Gao
2024-02-08 23:52:11

Right, the VM reboot stucks

naadir
2024-02-09 16:07:20

can't help massively but i think your VM is prompting for a passphrase on the console and you'll need to automate that in Packer with the keyboard commands prior to it being able to SSH.

:ty: Yongxiang Gao
Ryan Gough
2024-02-10 17:56:46

@Ryan Gough has joined the channel

Ryan Gough
2024-02-10 17:58:34

Really enjoying the image-builder project, but i’ve started to modify the packer-node.json files for my use-case, which doesn’t feel correct. Is there a way to skip the export of OVA to OVF, and just template instead of the post-processors?

Ryan Gough
2024-02-10 18:02:05

I can see it sort of was addressed here a few years back


but nothing ever became of it, it seems.

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-11 13:31:01

I’m not sure if this is what you’re asking exactly but rather than editing the existing files it’s possible to set the EXTRAPACKERVAR_FILES env var pointing to additional var files that layer on top of the built in ones.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-11 13:32:02

With regards to skipping the OVF. I’m not sure actually. I know when we build CAPV images we just ignore the other files and only copy what we need.

Ryan Gough
2024-02-11 13:34:21

Hi Marcus, i tried with copying packer-node.json, and using PACKERVARFILES but that didnt work, i’ll have a go with EXTRAPACKERVAR_FILESinstead?

Ryan Gough
2024-02-11 13:35:19

I only see two options really:

  1. Ignore them, like you suggested,

  2. Adjust the packer-node.json and remove the export definition in the clone block

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-11 13:47:52

I’m currently not at my laptop so can’t check but do you know if it’s possible to have packer not do the export? Is it something we could maybe have configured via env var or something? 🤔

Ryan Gough
2024-02-11 13:58:13

Doesn’t look like it; i mean the post-processing i thought about adding some env var logic on there, but the packer plugin doesn’t look like it’s able to disable the export option once set, i.e. enable true/false

😞 Marcus Noble
fad3t
2024-02-12 12:28:02

IIRC I had a look at something similar, and the current Packer config written in JSON comes with limitations

fad3t
2024-02-12 12:28:48

switching to HCL would allow to use dynamic blocks for example, to switch the export on/off in your case

fad3t
2024-02-12 12:28:52

but that's a lot of work 😕

fad3t
2024-02-12 12:29:51

I'm doing the same to add support for vApps, and for now I apply a patch to the packer-node.json file in the CI before calling the makefile. Dirty.. but it works 😛

Ryan Gough
2024-02-12 17:55:43

Yeah i think i’ll probably do this, at least it means maintaining a copy of the packer-node.json for now, but i see the only way this would work is if the vsphere packer plugin would support export.enable true/false

Lukas M
2024-02-11 13:23:16

@Lukas M has joined the channel

Slackbot
2024-02-11 13:24:11

This message was deleted.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-11 13:32:56

I don’t know much about kubevirt. Do you know what VM image type it needs? What makes you think that target isn’t enough?

Lukas M
2024-02-11 13:58:42

It should be QEMU, but with oem_id=kubevirt ( I guess ) to support and

Lukas M
2024-02-11 14:00:57

build-qemu-flatcar isnt enough, it does not support ignition, only coreos-cloudinit, that's not such a problem, but coreos-cloudinit does not handle # jinaja template string on the first line of userData generated by image-builder machinery

Lukas M
2024-02-11 14:04:37

I`ve tried qemu-flatcar target with OEM_ID=kubevirt, ..its there

test /home/core # cat /usr/share/oem/grub.cfg
set oem_id="kubevirt"
, but still not working..

Lukas M
2024-02-11 14:06:59

Hm, its probably not enough, , also flatcar.oem.id has to be set, not sure where exactly

Lukas M
2024-02-11 14:09:26

Its there also:

test /home/core # cat /proc/cmdline
rootflags=rw mount.usrflags=ro BOOT_IMAGE=/flatcar/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=7130c94a-213a-4e5a-8e26-6cce9662f132 rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 flatcar.first_boot=detected flatcar.oem.id=kubevirt verity.usrhash=d8aba28f890e180820484397bf8fd4ea722445662d25e7a2139360f12f74fa58

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-11 14:13:28

So the OEM is correctly being set but the images still aren’t working for you? What’s no working exactly? An error or just not able to boot at all?

Drew Hudson-Viles
2024-02-12 11:25:03

I haven't done the KubeVirt side of things with image builder just yet but plan on doing so this or next week for Ubuntu, so can't help too much on the Flatcar side but there is a kubevirt script in the qemu/scripts directory that may be of use?

Lukas M
2024-02-12 11:43:51

Its working with build-qemu-flatcar target, sorry for the fuzz...

Drew Hudson-Viles
2024-02-12 11:44:18

No worries!

Slackbot
2024-02-12 15:30:04

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-12 15:32:44

☝️ Agenda is currently empty. Is there anything anyone would like to discuss or should we skip?

mboersma
2024-02-12 15:58:42

I was thinking we should discuss EKS-Anywhere, but IDK what else there is to say once I thought about it: we just have to "return to sender" unless it's a bug reeproducible in image-builder on its own.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-12 15:59:14

Yeah lets do that!

👍 mboersma
mboersma
2024-02-12 15:59:45

Also I'm reworking the Azure pipelines to just be GH Actions and was wondering if that crossed a line or not. I'll put that on the agenda, should be a short discussion.

👍 Drew Hudson-Viles, Marcus Noble
Ryan Gough
2024-02-12 17:54:40

man i missed it!

mboersma
2024-02-12 19:59:16

There's a recording here:

YouTube
ogghead (https://www.youtube.com/@ogghead)
✅ Ryan Gough
Kazuya Nomura
2024-02-12 21:12:18

@Kazuya Nomura has joined the channel

Yongxiang Gao
2024-02-14 15:39:16

I see magic output image of ubuntu efi 2004 with partitions which doesn't honor preseed-efi.cfg

I used DEBUG=1 PACKER_LOG=1 to narrow down the partitions are generated at this step:

==> qemu: Pausing after run of step 'stepTypeBootCommand'. Press enter to continue. 
qemu: Not using a NetBridge -- skipping StepWaitGuestAddress

Do you know what else can we try (e.g. increase verbose level, etc) to narrow down further?

mboersma
2024-02-14 19:40:56

You can try adding PACKERFLAGS=-debug. Anything in $PACKERFLAGS gets passed to the packer build command, and -debug is suggested by .

developer.hashicorp.com
:ty: Yongxiang Gao
Makesh k
2024-02-16 09:27:34

@Makesh k has joined the channel

Ricky Sadowski
2024-02-20 14:02:19

@Ricky Sadowski has left the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-20 21:49:15

I’d like to get a new release put out as we’ve had a handful of fixes come in since the last. Is there any active PRs anyone would like to try to get in before I do?
I’m hoping to do a new release tomorrow if there’s no objections?

Drew Hudson-Viles
2024-02-20 21:50:16

Nothing from my side

mboersma
2024-02-20 22:54:13

Fine with me! There is little downside to doing frequent releases IMHO.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-20 22:58:13

Yeah. Just wanted to see if any PRs were ready for review before I do. 🙂
I mainly want to get the python fix for Azure Flatcar images out as we’re currently blocked by that at GS 😜

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-21 11:49:18

I'm kicking this off now. 🚀

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-21 12:03:04

Promo PR -

GitHub
Drew Hudson-Viles
2024-02-21 12:25:57

aaah I can't approve that one - thought I'd try for ya!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-21 12:26:20

Yeah, maintainers only 😞 Thanks for trying though 😄 I can wait for the others.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-21 12:36:47

Docs update for when the release is published (also include a shiny new script to automate most of this PR in the future 😉)

GitHub
🙌 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-21 14:50:14

Thanks y'all! 🙂

I'm still annoyed by the amount of manual steps needed to do a release. I really wish we could have the whole thing automated just from pushing a new tag. 🤔 Not sure how we'd handle the "wait for promo PR" and "wait for image being pullable" though.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-21 14:45:16

Image-builder v0.1.24 is now available:

Thanks to all contributors! 🎉 💙

GitHub
🎉 mboersma, Travis Holton
:thank_you: mboersma
Shashinandan Srinivasa
2024-03-06 18:05:26

Hi, sorry if my query looks novice. In the release.md file for v.0.1.24, why does it say the "current release of Image builder is v.0.1.23"?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-06 18:08:44

The release note and docs are updated after the release is available.

👍 Shashinandan Srinivasa
Hrvoje Bašić
2024-02-23 14:59:03

@Hrvoje Bašić has joined the channel

Slackbot
2024-02-26 15:30:08

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-26 16:31:01

☝️ Starting now 🙂 We've got a few items on the agenda for today.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-26 16:57:05

Huddle in thread about eks-anywhere 🧵

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-26 16:57:16

A huddle started

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-26 16:58:12

@Abhay Krishna Arunachalam

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-26 17:42:16

FYI - The general outcome with regards to image-builder was:
Regarding incoming issues - we’ll send people to the EKS-anywhere project (as they also want to be aware of issues with their CLI) and they will triage and open issues with us as and when needed. If we find this not manageable we can then look into having some of the EKS-anywhere members as a group within image-builder that we can assign issues to.
(cc @mboersma @jsturtevant @kiran keshavamurthy)

🙌 Abhay Krishna Arunachalam
:ty: jsturtevant
👍 kiran keshavamurthy, mboersma
orin
2024-02-26 17:11:16

@orin has joined the channel

Nicolò Ciraci
2024-02-27 10:51:25

@Nicolò Ciraci has joined the channel

Nicolò Ciraci
2024-02-27 10:52:52

Hello! I would like to start saying that I’m already sorry if this is not the correct way/channel to ask, but are other kinds of CRI runtime “supported”? I would like to build an image with CRI-O and PR the project. Thanks a lot!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-27 11:01:07

Only containerd is currently supported as installing it is currently one of the main tasks performed.
I suspect it might be possible to install CRI-O after image-builder has finished and switch to using that by default instead but as far as I know there's no way to configure image-builder to install a different CRI.

Nicolò Ciraci
2024-02-27 11:09:35

I was thinking about editing the ansible playbook to implement a ‘if-then’ logic to support other CRI flavour - this would be like implementing more optional variables and running the correct playbooks. Is this offlimits?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-27 11:16:08

Not at all. We'd welcome such a contribution. Just be aware that I think it would mean a lot of the vars files would need updating for all the providers to be able to support passing in the needed versions, sha's etc.

Nicolò Ciraci
2024-02-27 12:24:01

Ok, then I will work on this implementation and share the results. Hoping to be able to implement something quickly!

👍 Marcus Noble, jsturtevant
🤞 Marcus Noble
jsturtevant
2024-02-27 17:23:13

This should be possible to cofgirgure multi runtimes In a fairly clean way with ansible. on windows side we initially had two different runtimes, it's been removed now but was not to bad when it was done.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-02-27 17:51:35

Oh interesting. I wasn't aware of that. Might be worth taking a look through the git history to see how it was handled previously then.

Nicolò Ciraci
2024-02-27 17:57:12

Oh interesting , I’ve already implemented something and I will let you know tomorrow if it works as intended 😅 Should I refer to the ansible playbook for the older implementation?

jsturtevant
2024-02-27 18:52:04

Np, we can go with what you have and tweak from there

👍 Nicolò Ciraci, Marcus Noble
Nicolò Ciraci
2024-02-29 09:48:16

Hi! Right now I’m able to build a qemu backed image that leverage crio, I will implement gvisor for equal compatibility as containerd and then open a PR to get a review. If I can manage I will try also to prepare multiple version of builds to get a better coverage of os

👍 jsturtevant
Yongxiang Gao
2024-02-29 19:29:45

How to add a custom file to the image with image-builder code?

Abhay Krishna Arunachalam
2024-02-29 19:34:06

If this is related to using image-builder with EKS Anywhere, you can refer to this doc.

EKS Anywhere
:ty: Yongxiang Gao
Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-01 08:20:57

🤔 If we don't have that in base image-builder I think it would be a nice thing to add.

Abhay Krishna Arunachalam
2024-03-01 08:22:52

yeah I was thinking about that too. We use an additional role in the EKS-A repo for copying files.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-01 08:25:03

We have a similar one for loading additional executables from remote sources () but I think it would be nice to also have the same role you have.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-01 08:31:04

/cc @Drew Hudson-Viles you might be interested in this ☝️

Drew Hudson-Viles
2024-03-01 08:36:33

I'm not aware of any approach we have at the moment. It's certainly something we can look into though!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-01 08:43:54

I suspect we can likely copy over that role without much trouble.

👍 Drew Hudson-Viles
Yongxiang Gao
2024-03-03 18:07:10

Actually, I find a way to add with following:

  1. ansible/roles/sysprep/files

  2. ansible/roles/sysprep/tasks/

Drew Hudson-Viles
2024-03-04 09:25:15

That will work in a fork such as yours however it's not something that's easily adaptable for people using the core code so we will look into this.

Drew Hudson-Viles
2024-03-04 09:27:59

I've raised this here anyway and will look into this as soon as I can

GitHub
:thx_thanks: Marcus Noble
Abhay Krishna Arunachalam
2024-03-01 08:08:34

I noticed that a handful of ISO URLs in the Packer config files were returning 404s because of the images being removed from the mirror/release endpoint. I opened a PR to fix that and also update to latest point releases for some others. I was also thinking of switching the ubuntu 20.04 ISO URLs from the cdimage.ubuntu.com to the old-releases.ubuntu.com domain for consistency with ubuntu 22.04, any concerns with that?

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-01 08:22:01

This has been an ongoing pain 😞
Ubuntu doesn't include (or at least didn't used to) the latest release at the old-releases.ubuntu.com endpoint. So if we wanted to the latest release we needed to use the one that ended up breaking.

Abhay Krishna Arunachalam
2024-03-01 08:25:06

Yeah I have been thinking about how to solve this one too. But in this PR, I'm updating to the latest one available in old-releases, that should be okay right?

Abhay Krishna Arunachalam
2024-03-01 08:26:11

i'll update the checksums, thanks

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-01 08:26:15

Ah nice! I did have an issue for it 😆
Yeah, bumping the versions is always welcome, just remember that the checksums also need updating.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-01 08:26:20

ha, yeah

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-01 08:26:36

We do have an issue where we'd like to automate this but it's the checksums thats currently blocking that.

GitHub
Abhay Krishna Arunachalam
2024-03-01 08:26:45

i was wondering if we should have a periodic for updating ISO URLs

Abhay Krishna Arunachalam
2024-03-01 08:28:02

we could compute the checksums on the fly right? although it would take some time given the size of the images

Abhay Krishna Arunachalam
2024-03-01 08:28:35

or maybe parse the SHA256 file in the same releases endpoint

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-01 08:29:00

Hmm... that's not a bad idea. We might not actually have to calculate them... yeah, exactly that 😆 I feel like we're thinking the same

🙌 Abhay Krishna Arunachalam
Abhay Krishna Arunachalam
2024-03-01 08:39:13

Is there any reason not to switch from the ubuntu-legacy-server (available at cdimage.ubuntu.com) to the ubuntu-live-server (Available at old-releases.ubuntu.com) ISOs for ubuntu 20.04? I was thinking it'll be nice to standardize the ubuntu releases endpoints and not have separate sources?

Abhay Krishna Arunachalam
2024-03-01 08:43:09

also I think we can standardize URLs like these

and
I confirmed both are the same thing by checking the sha256 sum and the Entity tag (Etag) returned by curl , so we can use just one

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-01 08:43:38

I don't think so. I suspect its just a case of different people working on different areas leading to inconsistency

Abhay Krishna Arunachalam
2024-03-01 08:55:40

Makes sense, I will try to include that change in my PR

Abhay Krishna Arunachalam
2024-03-01 08:56:57

Also I think depending on the old-releases images isn't a bad thing, since they will get upgraded to the latest point release when image-builder runs the dist-upgrade step

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-01 09:10:00

True, didn't think about that.

Abhay Krishna Arunachalam
2024-03-02 00:27:01

Opened this PR to fix/update some ISO URLs and add a script for updating checksums in the future. Would appreciate some feedback on this. Thanks!

GitHub
Abhay Krishna Arunachalam
2024-03-02 00:35:28

Packer allows us to specify the iso_checksum as a URL pointing to a checksums file containing the actual checksum. I propose making that switch instead of hardcoding the checksum, to avoid having to update them each time. I saw we do it for Flatcar Linux, but we should ideally extend it for other OSs too.

I haven't made the proposed change in the PR, but I can update it if it sounds like a reasonable change.

👍 mboersma
RAKESH BOINAPALLY
2024-03-04 18:31:11

Quick question i'm running into following error did any one face this error in the past any lead

Build 'vsphere-iso.vsphere' errored after 17 minutes 57 seconds: error exporting vm: ServerFaultCode: Permission to perform this operation was denied.

Abhay Krishna Arunachalam
2024-03-04 18:52:55

error exporting vm
so is this occuring at the end of the image build?

Could you verify that the vsphere user has these privileges?

RAKESH BOINAPALLY
2024-03-04 19:12:03

End of the image

RAKESH BOINAPALLY
2024-03-04 19:33:06

it is failing at this particular step ==> vsphere-iso.vsphere: Goss validate ran successfully
==> vsphere-iso.vsphere:
==> vsphere-iso.vsphere:
==> vsphere-iso.vsphere:
==> vsphere-iso.vsphere: Downloading spec file and debug info
vsphere-iso.vsphere: Downloading Goss specs from, /tmp/goss-spec.yaml and /tmp/debug-goss-spec.yaml to current dir
==> vsphere-iso.vsphere: Executing shutdown command...
==> vsphere-iso.vsphere: Deleting Floppy drives...
==> vsphere-iso.vsphere: Deleting Floppy image...
==> vsphere-iso.vsphere: Eject CD-ROM drives...
vsphere-iso.vsphere: Starting export...
==> vsphere-iso.vsphere: Provisioning step had errors: Running the cleanup provisioner, if present...
==> vsphere-iso.vsphere: Clear boot order...
==> vsphere-iso.vsphere: Power off VM...
==> vsphere-iso.vsphere: Destroying VM...
Build 'vsphere-iso.vsphere' errored after 17 minutes 33 seconds: error exporting vm: ServerFaultCode: Permission to perform this operation was denied.

==> Wait completed after 17 minutes 33 seconds

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso.vsphere: error exporting vm: ServerFaultCode: Permission to perform this operation was denied.

RAKESH BOINAPALLY
2024-03-05 21:02:27

it was a permission issue things did go fine

RAKESH BOINAPALLY
2024-03-05 21:02:33

after fixing the permission issue

Abhay Krishna Arunachalam
2024-03-05 22:26:45

gr-nice-2

snevedomski
2024-03-05 16:14:53

@snevedomski has joined the channel

snevedomski
2024-03-05 16:52:56

Hi all. Need your help with proxmox image-builder.
Followed this instruction:
But I'm getting "write tcp 192.168.30.33:59042->192.168.30.2:8006: write: broken pipe" error.

And I don't even get how it is supposed to work. Instruction doesn't provide password or token secret. Tried adding them as env vars. Nothing changes.
What am I doing wrong?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-05 17:06:08

@mcbenjemaa are you able to help here?

👍 snevedomski
👍:skin_tone_3: mcbenjemaa
mcbenjemaa
2024-03-05 19:02:57

As i can see, this is network issues happens in your setup.

You will need to rerun or you can actually change the values so you can use an existing ISO.

snevedomski
2024-03-05 20:06:01

Thank you. I'll try it.

snevedomski
2024-03-05 22:31:05

Rerun didn't help
But I figured out my problem.
PROXMOXUSERNAME is proxmox token ID
PROXMOX
TOKEN is proxmox token secret
This is very counterintuitive.

Thanks everyone for help

👍:skin_tone_3: mcbenjemaa
Abhay Krishna Arunachalam
2024-03-05 22:47:07

You can refer to the Packer Proxmox builder docs for the syntax and semantics of the username and token fields

developer.hashicorp.com
👍 snevedomski
mcbenjemaa
2024-03-07 11:58:58

Oh, yeah exactly.
Packer plugin uses Username/password and token
But in capi provider we only use Token based authentication.

RAKESH BOINAPALLY
2024-03-06 18:15:58

Hi All did any one use govc to export ova from one vcenter locally then import it to different vcenters.It would be very helpful if some has done this kind of stuff so that dont need to reinvent the wheel.I'm trying to use govc to export and import ova created using image builder

Shashinandan Srinivasa
2024-03-06 21:37:51

gr-wave_animated the last image-builder version that works in my CI to build custom vsphere images is v0.1.19. Even the latest one (v0.1.24) just hangs at "Waiting for ssh to be available". Any pointers to troubleshoot this behavior?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-06 21:45:29

If 1.19 is the last that worked for you then I suspect something in this release broke for you: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.20

I see a couple vsphere changes there that might give you some insight into what might be wrong.

It also might be useful if you share what make target you’re using, what vars you’re providing and if you’re using the container image to run image-builder or not.

I can confirm that, at least for me, the latest release was able to successfully build a flatcar image for vsphere.

GitHub
Shashinandan Srinivasa
2024-03-06 22:02:47

"# make build-node-ova-vsphere-ubuntu-2204" Its a gitlab pipeline with following steps

Shashinandan Srinivasa
2024-03-06 22:02:54

I clone the tag, in this case v0.1.19 -> add custom ansible roles -> build docker image -> run the make target using the new built docker image

Shashinandan Srinivasa
2024-03-06 22:05:00

I will try using the docker image that you have directly and get back. Its flaky for sure. I have tried running this directly on host, without any container. Sometime it works and sometimes it doesn't

Shashinandan Srinivasa
2024-03-06 22:48:37

From the VM console, this is where it stops and doesn't proceed. On the cli its waiting for ssh

Shashinandan Srinivasa
2024-03-06 22:48:41

*Thread Reply:* None

Shashinandan Srinivasa
2024-03-06 23:16:19

So, direct checkout to host and running it works.

Shashinandan Srinivasa
2024-03-06 23:17:32

If I try to create a custom docker image and run make target from a container out of it, thats when it hangs waiting for ssh. Any setting I am missing on the docker host or docker build that will help here?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-07 06:51:13

Are you building your docker image on top of ours and then just adding in the ansible roles you need? Or are you building from scratch?

Shashinandan Srinivasa
2024-03-11 18:12:13

Building from scratch and I have used your Dockerfile as reference. Only difference is that I clone specific tag. "git clone --depth 1 --tags v0.1.24 --single-branch"

Shashinandan Srinivasa
2024-03-11 18:13:47

I removed the above step of cloning just a single tag and cloned entire repo. The build happens without issues. It takes some time, but it does build successfully.

Shashinandan Srinivasa
2024-03-11 18:14:24

I will update when I find more details about this.

Mitchel Haring
2024-03-07 21:57:31

Any thoughts on adding Rocky 9 Linux for OVA? -

GitHub
👍 mboersma
Yongxiang Gao
2024-03-10 08:31:28

How to override kubernetes version in the output image, I know with image-builder command, we can use option "release-channel" as follows:

image-builder build --os ubuntu --os-version 20.04 --hypervisor baremetal --release-channel 1-28 --firmware efi
How about the command "make build-raw-ubuntu-2004-efi" from the source of ?

Yongxiang Gao
2024-03-10 08:36:33

With this prefix?

PACKER_FLAGS="--var 'kubernetes_rpm_version=1.28.3' --var 'kubernetes_semver=v1.28.3' --var 'kubernetes_series=v1.28' --var 'kubernetes_deb_version=1.28.3-1.1'"

Drew Hudson-Viles
2024-03-10 08:37:09

Hi,

If you look in the packer.json file, you'll be able to see a bunch of variables you can override.

For example the Kubernetes version is overridden using this one: https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/packer/raw/packer.json#L164.

In your variables file, you can add it and it will override it.
https://github.com/kubernetes-sigs/image-builder/blob/main/docs/book/src/capi/capi.md?plain=1#L97

You can also override them directly with flags.
https://github.com/kubernetes-sigs/image-builder/blob/main/docs/book/src/capi/capi.md?plain=1#L91

:ty: Yongxiang Gao
mboersma
2024-03-11 14:51:33

Reminder: Image-Builder office hours start in 40 minutes. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-11 14:55:03

Oh... we're in the fun two weeks of DST mis-match 😆

mboersma
2024-03-11 14:56:22

Oh...yikes. Should we go with Pacific time or wait an hour?

mboersma
2024-03-11 14:56:54

I wondered why I didn't see Slackbot, maybe we should wait an hour?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-11 14:56:55

My calendar entry is correct (in 40 min) but it's an hour earlier than it normally is for me.
The time still works for me though.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-11 14:57:17

Although the doc does say UTC thinking

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-11 14:57:32

Oh wait, that's right

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-11 14:57:45

3:30pm UTC is in ~30 min

mboersma
2024-03-11 14:58:17

I am very much a morning person, but the Monday when DST changes just sucks. 🙂 Ok, see you in 30 minutes.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-11 14:58:35

I just suck at timezones in general 😆

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-11 14:58:55

Looks like when I set up the slack reminder it was during DST

👍 mboersma
Slackbot
2024-03-11 15:30:15

Reminder: Image-Builder office hours start in 1 hour. Agenda:

😀 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-11 15:30:51

☝️ Ignore, it's actually now

giridhar M Sharma
2024-03-11 22:01:28

@giridhar M Sharma has joined the channel

giridhar M Sharma
2024-03-11 22:08:43

Hello All,
we are trying to build a ubuntu 22.04 on vsphere 8.
I attended the office hours meeting today and discussed this problem. There were few threads which were suggested. I tried everything but still it doesn't work.
Threads I went through



The first one seems somewhat different problem. Second thread matches the problem I am facing. There was a suggestion to make changes to the bootcommandprefix in one of the workarounds. With this at-least the ssh is becoming available (manually), however, make still getting stuck at Waiting for SSH to become available... . If I go and look at the console, it would be stuck in choose the language screen.

I am just thinking, is the boot command for ubuntu22.04 correct or something is missing here.
boot command for 22.04 given in packer/ova/ubuntu-2204.json is,

"boot_command_prefix": "clinux /casper/vmlinuz ipv6.disable={{ user boot&#95;disable&#95;ipv6 }} --- autoinstall ds='nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/22.04/'initrd /casper/initrdboot",



#1076 build-qemu-ubuntu-2204 stuck in "Waiting for SSH to become available..."
kubernetes-sigs/image-builder | Feb 15th, 2023 | Added by GitHub


#1301 [ova] ubuntu22 won't build on vsphere 7.x
kubernetes-sigs/image-builder | Sep 20th, 2023 | Added by GitHub

GitHub
GitHub
GitHub
snevedomski
2024-03-11 22:41:24

That's what I use.
"clinux /casper/vmlinuz --- autoinstall ds='nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/22.04/'initrd /casper/initrd boot "
Works well for me.

snevedomski
2024-03-11 22:46:36

Your boot command is probably fine too.
You get language screen after some time if you cannot establish SSH connection or if you boot from install disk after reboot.
Check SSH related variables and boot order

giridhar M Sharma
2024-03-11 22:49:50

Did you happen to do any edits in the image builder configs to make this work? The same vpshere is building 2004 ova just fine.

snevedomski
2024-03-11 23:02:31

nope, But I'm using proxmox image-builder

giridhar M Sharma
2024-03-11 23:12:26

ah ok. So this issue is something specific to vsphere then

giridhar M Sharma
2024-03-12 19:06:58

Hi @snevedomski: Till I get some solution on the vsphere, I thought I will give it a shot with proxmox as it is working for you as is. I created a VM out of the published ISO. However when I run the build command, I hit this issue and I have no idea why its failing. Any hints here will be of great help

==> proxmox-iso.ubuntu-2204: Post "https://:8006/api2/json/nodes/pve/storage/local/upload": write tcp :58174->proxmox_vm_ip:8006: write: broken pipe
Build 'proxmox-iso.ubuntu-2204' errored after 4 seconds 920 milliseconds: Post "https://:8006/api2/json/nodes/pve/storage/local/upload": write tcp :58174->10.206.140.162:8006: write: broken pipe

snevedomski
2024-03-13 01:03:07

try to connect to proxmox server from terminal.

Justinas B
2024-07-08 10:04:09

Hey @snevedomski , is it still working for you?

Justinas B
2024-07-08 10:04:32

my build for proxmox also gets stuck on language selection screen and i cannot find a way around it

Ahree Hong
2024-03-13 17:41:45

@Ahree Hong has joined the channel

Ahree Hong
2024-03-13 18:14:03

Hello all!
I opened a PR to upstream a patch we added in eks-anywhere which adds qemu support for RHEL9

GitHub
👍 mboersma
:thank_you_icon: Marcus Noble
Abhay Krishna Arunachalam
2024-03-13 18:16:47

Hiya @Ahree Hong!gr-wave_animated

Drew Hudson-Viles
2024-03-13 20:55:29

Image-builder v0.1.25 is now available:
Thanks to all contributors! 🎉

GitHub
:parrotk8s: Marcus Noble, Vignesh Goutham, Mitchel Haring, Abhay Krishna Arunachalam, jsturtevant
Vignesh Goutham
2024-03-13 21:35:38

Hello everyone, inspired from @Ahree Hong’s contribution, a PR to support Raw RHEL-9 both bios and efi. Please take a look.

GitHub
:thank_you_icon: Marcus Noble
👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-14 08:30:50

😞 We really need to come up with a good solution for the ubuntu ISO URLs.
Another instance of confusion regarding using the old url instead of latest -

Anyone have any ideas on how we can have the URL static but still use the latest available? I'm surprised Ubuntu doesn't provide this.

Abhay Krishna Arunachalam
2024-03-14 08:47:10

Really not loving this hacky approach but it does give the latest available one on the releases endpoint

$ curl -L  | grep -o 'href="ubuntu-22.04.-live-server-amd64.iso">' | gsed -e "s/href=\"//g" | gsed -e 's/">//g' | uniq
ubuntu-22.04.4-live-server-amd64.iso

$ curl -L | grep -o 'href="ubuntu-20.04.
-live-server-amd64.iso">' | gsed -e "s/href=\"//g" | gsed -e 's/">//g' | uniq
ubuntu-20.04.6-live-server-amd64.iso
we could get the value dynamically and jq it into the packer config file before the build

Abhay Krishna Arunachalam
2024-03-14 08:48:25

Keep in mind, this only addresses the issue of URL availability, not build reproducibilty.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-14 08:48:39

But that URL will eventually stop working which would mean image-builder breaks without changes 😞

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-14 08:48:49

Ha, yeah exactly 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-14 08:50:03

What I really want is for Canonical to provide an endpoint that we can give a version to and it redirects to the appropriate URL.

✅ Abhay Krishna Arunachalam
Abhay Krishna Arunachalam
2024-03-14 08:51:06

But that URL will eventually stop working
you mean the ISO URL will return 404? or it will point to a bad ISO?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-14 08:51:36

Return 404 once a new update is available.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-14 08:51:50

The releases.ubuntu.com only makes the latest available

Abhay Krishna Arunachalam
2024-03-14 08:52:46

The above curl command bypasses that by getting the directory listing at the time of build (rather just before), and setting the ISO URL to that.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-14 08:53:08

Oh you mean at time of running the make targets

Abhay Krishna Arunachalam
2024-03-14 08:53:20

we're querying so if it gets updated to 22.04.5, it would still get that

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-14 08:53:55

But then we have the problem that the same version of image-builder could introduce new changes because a new Ubuntu version was release. 😞

Abhay Krishna Arunachalam
2024-03-14 08:54:07

in that case, we don't have the issue of 404, but image-builder would still break if the new ISO is broken

Abhay Krishna Arunachalam
2024-03-14 08:54:11

yup

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-14 08:55:37

I might see if I can find a contact at Canonical to discuss it with them 🤔

🙌 Abhay Krishna Arunachalam
Abhay Krishna Arunachalam
2024-03-14 09:01:10

we could still pin to the latest releases version (possibly in a Makefile or tag files like we use in EKS Anywhere) but have some automation around it that checks for the latest releases periodically and updates the pinned version. Then atleast we have the auditability.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-14 09:03:07

Looks like Canonical have a booth at KubeCon next week so I might try and see if I can find someone appropriate to talk with.
Failing that, I might see if we could get a proxy service hosted in the Kubernetes community cluster to handle this.

Abhay Krishna Arunachalam
2024-03-14 09:04:42

Nice! I was looking into other mirrors too, but no one seems to have the combination of old and new releases images (why would they, I guess)

mboersma
2024-03-14 15:40:37

Failing that, I might see if we could get a proxy service hosted in the Kubernetes community cluster to handle this.
That was the only solution I could think of here--maintain our own redirect URLs. It would put a maintenance burden on the image-builder team, but at least we wouldn't have to update code every time it changes.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-14 15:41:49

If it's something we can run on the community cluster it should be a fairly simple application. But I'm much prefer Canonical actually did it rather than us if possible

👍 mboersma, Abhay Krishna Arunachalam
Abhay Krishna Arunachalam
2024-03-18 17:02:18

Hi all, looking for an approver to approve this.

GitHub
Abhay Krishna Arunachalam
2024-03-18 18:36:12

Hello all, I opened this PR to move the ks configs for RHEL raw builds inside the http directory since that's what Packer expects when serving these to the VM

👍 mboersma
Abhay Krishna Arunachalam
2024-03-19 16:17:02

cc @mboersma @Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-19 16:20:17

Sorry, I saw your PRs but I’m away at Kubecon this week and haven’t had a chance to take a look yet. @Drew Hudson-Viles maybe able to take a look?

Abhay Krishna Arunachalam
2024-03-19 16:38:31

No worries, apologies for the noise. Wanted to get these two merged because they could be potential blockers for image builder users

:thank_you_icon: Marcus Noble
Drew Hudson-Viles
2024-03-19 16:39:42

I'll be able to take a look shortly, no problem

:ty: Abhay Krishna Arunachalam
galop
2024-03-19 08:42:25

@galop has joined the channel

rodrigodelmonte
2024-03-19 22:53:32

@rodrigodelmonte has left the channel

Abhay Krishna Arunachalam
2024-03-19 23:25:33

Opened a cleanup PR for symlinks and unused kickstart files

GitHub
RAKESH BOINAPALLY
2024-03-21 16:27:33

i was running image-builder with replacing old-ubuntu-releases with seems like at vm level in vsphere it is asking to make choice english and presss enter did any one ran into such issue

releases.ubuntu.com
Abhay Krishna Arunachalam
2024-03-21 17:16:12

If it's prompting you then it means the Packer VM did not get the autoinstall configuration and is dropping into the interactive Ubuntu install, which is not desired.

RAKESH BOINAPALLY
2024-03-21 17:56:05

that is strange all i was doing is passing ubuntu iso check sum and url as environment variables

RAKESH BOINAPALLY
2024-03-21 20:29:09

Also im noticing Buffer I/O error on disk dev fd0

RAKESH BOINAPALLY
2024-03-21 20:29:27

which seems to be using floppy disk in vsphere any option to avoid floppy disk

RAKESH BOINAPALLY
2024-03-22 00:51:56

Seems like this is the issue that i'm running into

GitHub
RAKESH BOINAPALLY
2024-03-22 18:46:06

any ideas folks

RAKESH BOINAPALLY
2024-03-22 18:46:22

i tried cd_files option still it is asking for english language selection

Abhay Krishna Arunachalam
2024-03-22 19:47:51

@mboersma could you give me an /approve here?

GitHub
👍 mboersma
Slackbot
2024-03-25 15:30:26

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles
2024-03-25 15:32:18

I'm afraid I'll be unable to make this one due to a house viewing I need to attend.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-25 15:44:07

The agenda is currently empty so I suggest we skip this one then. (I'm also not feeling too great so happy to go rest instead 😛 )

👍 Drew Hudson-Viles, mboersma
Drew Hudson-Viles
2024-03-25 16:00:57

Works for me. Feel better buddy!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-25 16:03:16

Thanks 🙂 Pretty sure its just post-conference grossness 😛

Drew Hudson-Viles
2024-03-25 16:05:44

hahah probably!

mboersma
2024-03-25 16:11:20

Feel better Marcus! See you both in a couple weeks.

👍 Drew Hudson-Viles
jsturtevant
2024-03-25 16:42:35

cone there is a PR to remove Virtual Box support. It was used by sig-windows for awhile but is not longer being used or updated. In effort to clean up image-builder we will be removing it unless anyone objects. Will release the PR in 2 weeks. Thanks! cc: @Amim Knabben

👍 Amim Knabben, Marcus Noble, Drew Hudson-Viles, mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:14:49

Those building images should be aware of this and check your recent builds to see if they’re vulnerable.

Version 5.6.0 and 5.6.1 of xz contain a backdoor aimed to bypass SSH authentication: https://lwn.net/Articles/967180/

lwn.net
Drew Hudson-Viles
2024-03-29 20:15:28

Thanks for posting this - I was going to get around to it shortly but childcare is a thing right now 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:16:25

Yeah. I haven’t looked into the details too much yet as I’m not at a laptop but I know at least the latest stable Flatcar version is unaffected. Not sure about other distros though.

Drew Hudson-Viles
2024-03-29 20:17:49

My understanding of the read I had was that it's unlikely to affect most distros due to it being in newer/pre-release builds, but that doesn't necessarily mean everyone is fine so they should check.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:19:59

But image builder performs an update of all packages as post of the build process

Drew Hudson-Viles
2024-03-29 20:21:21

It does indeed. I guess if it's in the package-manager - which seems possible, it could be an issue. I'm going to build a ubuntu 22.04 shortly and check it out anyway

👍 Marcus Noble
Abhay Krishna Arunachalam
2024-03-29 20:35:07

I have a cluster running Ubuntu 20.04.6 nodes built with image-builder and here's my output:

/# xz --version
xz (XZ Utils) 5.2.4
liblzma 5.2.4

Abhay Krishna Arunachalam
2024-03-29 20:35:44

I think this was built March 7

Drew Hudson-Viles
2024-03-29 20:36:16
22.04.4 LTS

xz --version
xz (XZ Utils) 5.2.5
liblzma 5.2.5
Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:36:21

20.04 might not have been updated

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:36:29

Oh nice!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:37:02

So at least for Ubuntu as long as people aren’t manually updating packages they should be safe

Drew Hudson-Viles
2024-03-29 20:37:17

I'm pretty certain in that long thread it did mention it was pre-release builds getting it. I need to digest it properly as it was a skim whilst watching In The Night Garden 😄

Drew Hudson-Viles
2024-03-29 20:39:05

I've just updated the package on my bare metal node at home which also runs 22.04 and it seems it's not getting that compromised package (yet).

apt-cache madison xz-utils
xz-utils | 5.2.5-2ubuntu1 | jammy/main amd64 Packages

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:40:41

Ok. So Flatcar and Ubuntu seem safe.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:41:01

I know Archlinux has been vulnerable but that’s not something we need to worry about

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:41:16

Not sure about RHEL or CentOS

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:44:29

From RedHat:

Fedora Linux 40 users may have received version 5.6.0, depending on the timing of system updates. Fedora Rawhide

redhat.com
Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:45:17

From Debian:

Right now no Debian stable versions are known to be affected.
Compromised packages were part of the Debian testing, unstable and
experimental distributions, with versions ranging from 5.5.1alpha-0.1
(uploaded on 2024-02-01), up to and including 5.6.1-1.

Drew Hudson-Viles
2024-03-29 20:45:33

phew

Drew Hudson-Viles
2024-03-29 20:45:46

Pretty clear on the whole then. Maybe a couple edge cases

Marcus Noble (k8s@marcusnoble.co.uk)
2024-03-29 20:47:40

I’m not sure about CentOs, Photon or RockyLinux but I’d guess they’re also ok as I don’t think any of those run with bleeding edge packages.

saifeddine Rajhi
2024-03-31 18:44:55

@saifeddine Rajhi has joined the channel

Staerion
2024-04-01 09:06:44

@Staerion has joined the channel

Karine Santos
2024-04-01 22:45:30

Hello everyone!
I'm looking for the image builder code where these steps for creating vms and resources are defined, and I can't find it. Can someone help me?

Abhay Krishna Arunachalam
2024-04-01 22:54:55

Image-builder Make targets just call the packer build command with the appropriate configuration files/values.

Packer is in charge of creating the VMs for each virtualization platform (vsphere/openstack/ami/nutanix, etc). through its plugins. In this case, the platform is Openstack so the code for creating VMs and resources will be located at . Plugins directly interact with the virtualization platforms through the available SDKs and APIs (in this case, )

Karine Santos
2024-04-02 00:25:54

tks @Abhay Krishna Arunachalam

Karine Santos
2024-04-02 00:28:39

hello everyone!
Could you help me with a question? I'm trying to upload some k8s images, and I'm encountering the following error. Is there a way to find out why the process of detaching the volume from the instance is not occurring?

jsturtevant
2024-04-02 16:49:31

which provider is this? It looks to be an issue with the provider code

Karine Santos
2024-04-02 17:27:29

hello @jsturtevant
We use the openstack provider. I would need to increase the server state retry (line 5013 to 5016). For the case below, we understand that we need more time for the desired state. It's possible?

Karine Santos
2024-04-02 17:29:31

The desired behavior is this (here it tries at least 5x):

Karine Santos
2024-04-02 17:30:20

My question is if I can somehow change (via ansible, etc.) the retry of the instance state

jsturtevant
2024-04-03 18:35:49

Ansible does have a retry on the tasks: would that help?

Kepler SysAdmin
2024-04-05 12:15:59

oh, we kind of are in the same page
what it is happening in my own opinion
the image builder starts to upload the image, but it jumps to fast for the next step
creating/saving/upload an image could take even 10 mins

Kepler SysAdmin
2024-04-08 13:18:40

@Karine Santos
Did you find a solution for this?

Drew Hudson-Viles
2024-04-08 16:02:02

I don't have a solution for this but I can confirm I use the OpenStack builder on an almost weekly basis without any issues.

It looks like your error is similar to what is being seen here:

and here:


How is OpenStack created in your case - if it's using the methods defined above, you may be hitting this bug.

GitHub
Launchpad
Karine Santos
2024-04-16 23:25:32

Hello everyone! Thank you for all the contributions!
@Kepler SysAdmin This issue has been fixed in the packer plugin 🙌:
BUGFIX:
RELEASE: V.1.1

If you use packer-plugin-openstack, you will need:
1. Add to the image builder to use a specific version of the packer plugin here:

2. Add variable in dependencies to use this config.pkr.hcl file:

GitHub
GitHub
GitHub
GitHub
👍 Kepler SysAdmin
Karine Santos
2024-04-16 23:27:30

By adjusting this, I was able to upload the images!!!

👍 Drew Hudson-Viles
jhon
2024-04-02 16:33:44

@jhon has joined the channel

Danny Bessems
2024-04-03 06:17:14

Hello, is it possible to easily patch the vsphere ova builders to include an extra 1MB disk when building within the cluster-node-image-builder-amd64 container? So that when deploying with VSphereMachineTemplate I can use spec.template.spec.additionalDisksGiB[] straight away, without first having to edit the template through vCenter...

Danny Bessems
2024-04-03 06:52:49

I know that it is defined within packer-node.json; but I´d rather not override the entire file if possible.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 09:16:58

You should be able to provide additional vars files that overlay onto the existing ones in image-builder.
If you mount a json file with the vars you need into the container you can then specify the PACKERVARFILES environment variable to point to that vars file.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 09:14:23

I've made some progress on the issue we were having with Ubuntu ISO URLs 🎉
It turns out that Ubuntu does have some stable URLs that redirect to old-releases in some situations where needed. I've updated the issue with all the details and I've opened the PR below to update all the ISO URLs we currently reference 🙂 (Except for Ubuntu 23.04 as that currently doesn't have the stable URLs while still in beta 😒)

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 09:22:19

@Abhay Krishna Arunachalam That update-iso-checksums.sh is fantastic 😄 Made things so easy!

Abhay Krishna Arunachalam
2024-04-05 09:35:04

Glad it's finding some use!

For the Ubuntu 20.04 ones, since we're moving from the legacy server install to the live server install, we need to change the boot command and switch from the debian-installer (d-i) to the subiquity autoinstaller (similar to the 22.04 ones).

Abhay Krishna Arunachalam
2024-04-05 09:40:13

I hadn't included the above boot command and other changes in my original PR that updated the URLs, and it broke our builds when we started consuming IB v0.1.25 in EKS-A, as the VM couldn't find the kickstart file.

So I set out to try and add the necessary changes as a patch on EKS-A, with the intention of contributing the change upstream. I experimented with every combination of boot command sequence possible but couldn't figure it out for the life of me, so ended up reverting the URL changes to Ubuntu 20.04 alone.

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 09:44:19

Ah! I did wonder about those 20.04 ones and if they were different. I thought it might be just different host they were pulled from, didn't realise the actual images were different.
I'll revert those ones and just tackle 22.04 for now.

Abhay Krishna Arunachalam
2024-04-06 01:12:08

Sounds good! Though it'd be nice to eventually align with 22.04 and get rid of the preseed approach entirely as it's been deprecated since 20.04 came out

:plus1: Marcus Noble
Kepler SysAdmin
2024-04-05 09:55:25

@Kepler SysAdmin has joined the channel

Kepler SysAdmin
2024-04-05 10:00:06

Hello there,
I am trying building images, following the instructions on:
but failing on: openstack: Error waiting for image: Resource not found
logs:

==> openstack: Downloading spec file and debug info
openstack: Downloading Goss specs from, /tmp/goss-spec.yaml and /tmp/debug-goss-spec.yaml to current dir
==> openstack: Stopping server: e8c26074-6928-4ad7-9d6e-562b63501c19 ...
openstack: Waiting for server to stop: e8c26074-6928-4ad7-9d6e-562b63501c19 ...
==> openstack: Terminating the source server: e8c26074-6928-4ad7-9d6e-562b63501c19 ...
==> openstack: Creating the image: ubuntu-2204
openstack: Image: c5bc4f59-2440-4094-9720-fa06ae5802b5
==> openstack: Waiting for image ubuntu-2204 (image id: c5bc4f59-2440-4094-9720-fa06ae5802b5) to become ready...
==> openstack: Error waiting for image: Resource not found
==> openstack: Provisioning step had errors: Running the cleanup provisioner, if present...
==> openstack: Deleted temporary floating IP '71e76a4a-a6e6-4cb2-bc98-9e7c5c6362d3' (178.73.197.24)
==> openstack: Terminating the source server: e8c26074-6928-4ad7-9d6e-562b63501c19 ...
==> openstack: Error terminating server, may still be around: Resource not found
==> openstack: Deleting volume: fdb85e58-1c8a-4d52-bdf2-3fa6720e9b9a ...
==> openstack: Deleting temporary keypair: packer_660f9e6a-fe25-d9cf-6c76-4c2f0149e5d4 ...
Build 'openstack' errored after 10 minutes 8 seconds: Error waiting for image: Resource not found
and the var_file.json I am using:
{
"source_image": "",
"networks": "",
"flavor": "",
"floating_ip_network": "public",
"image_name": "ubuntu-2204",
"image_visibility": "public",
"image_disk_format": "raw",
"volume_type": "",
"ssh_username": "ubuntu",
"kubernetes_version": "1.28.7"
}
be aware the instance, volume and ssh key gets create it, where it fails is on:
==> openstack: Waiting for image ubuntu-2204 (image id: c5bc4f59-2440-4094-9720-fa06ae5802b5) to become ready...
==> openstack: Error waiting for image: Resource not found
I tried from my laptop, and later from a node that is in the same physical network than the OpenStack Cloud

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 10:20:47

I've not used OpenStack before but are you able to see image c5bc4f59-2440-4094-9720-fa06ae5802b5 in your OpenStack cloud UI at all?

Kepler SysAdmin
2024-04-05 10:24:15

  1. the image isn't there

  2. The time pass so fast, that I am unable to catch it

I am thinking i need to modify some timing between Waiting for image till the next step, because if the image is uploading or creating from a volume, it can last some time around 5-10 mins

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 10:27:37

You could try setting the env var PACKER_LOG=1 to see if you get any more info from the verbose log output.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 10:28:19

Is it possibly a permissions problem? The credentials image-builder running with not having permission to create images? (I'm just guessing here)

Kepler SysAdmin
2024-04-05 10:28:50

the credentials have the right to create images

Kepler SysAdmin
2024-04-05 10:29:01

but I will try: PACKER_LOG=1

🤞 Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 10:31:11

Is it possible you're facing this issue:

Any errors in the glance api?

Stack Overflow
Kepler SysAdmin
2024-04-05 10:32:35

thx, I need to dig into this, once I found a solution, I will let you know
Thank you again

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 10:32:53

No worries 🙂 Hope you manage to track it down

Kepler SysAdmin
2024-04-05 10:51:51

logs:

# kubectl -n openstack logs pod/glance-api-76759946d4-g9d4n -f | grep 72a7187e-43fa-425b-a52a-a56489e04e6d
Defaulted container "glance-api" out of: glance-api, init (init), glance-perms (init), ceph-keyring-placement (init)
[pid: 14|app: 0|req: 6057/48488] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr 5 09:39:31 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 44 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
[pid: 7|app: 0|req: 6058/48489] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr 5 09:39:33 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 44 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
[pid: 10|app: 0|req: 6067/48490] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr 5 09:39:35 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 53 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
[pid: 8|app: 0|req: 6057/48493] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr 5 09:39:40 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 35 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
2024-04-05 09:39:42.286 13 ERROR glance_store._drivers.rbd [None req-65915cac-69f6-45bf-9c50-891063151686 8000aba2f7aa49b6a8d15a44a39f9eb5 ca09816bc9c148cc8e8f79af1068db97 - - default default] Failed to store image 72a7187e-43fa-425b-a52a-a56489e04e6d Store Exception unable to receive chunked part: OSError: unable to receive chunked part
[pid: 13|app: 0|req: 6063/48494] 10.0.1.213 () {40 vars in 1101 bytes} [Fri Apr 5 09:39:42 2024] PUT /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d/file => generated 228 bytes in 582 msecs (HTTP/1.1 500) 4 headers in 184 bytes (1 switches on core 0)
[pid: 11|app: 0|req: 6064/48495] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr 5 09:39:42 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 45 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
[pid: 14|app: 0|req: 6058/48496] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr 5 09:39:44 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 957 bytes in 31 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
[pid: 10|app: 0|req: 6068/48498] 188.78.244.135 () {44 vars in 910 bytes} [Fri Apr 5 09:39:47 2024] GET /v2/images/72a7187e-43fa-425b-a52a-a56489e04e6d => generated 139 bytes in 16 msecs (HTTP/1.1 404) 4 headers in 164 bytes (1 switches on core 0)

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 10:52:31

Failed to store image 72a7187e-43fa-425b-a52a-a56489e04e6d Store Exception unable to receive chunked part: OSError: unable to receive chunked part
What an incredibly unhelpful error 😞

Kepler SysAdmin
2024-04-05 10:53:04

just changing the logs from info to debug, and run the image-build again

Kepler SysAdmin
2024-04-05 10:53:14

that will take a couple of mins

Kepler SysAdmin
2024-04-05 11:39:16

2024-04-05 10:33:26.737 9 ERROR glance.common.wsgi OSError: unable to receive chunked part

Kepler SysAdmin
2024-04-05 11:42:48

if you wonder, this is after:

[pid: 9|app: 0|req: 24/165] 10.0.1.213 () {40 vars in 1100 bytes} [Fri Apr  5 10:33:26 2024] PUT /v2/images/796d1a83-a325-43af-9c95-f1e098f1467b/file => generated 228 bytes in 636 msecs (HTTP/1.1 500) 4 headers in 184 bytes (1 switches on core 0)
/var/lib/openstack/lib/python3.10/site-packages/pycadf/identifier.py:71: UserWarning: Invalid uuid: unknown. To ensure interoperability, identifiers should be a valid uuid.
warnings.warn(('Invalid uuid: %s. To ensure interoperability, '
[pid: 14|app: 0|req: 21/166] 188.78.244.135 () {44 vars in 909 bytes} [Fri Apr 5 10:33:27 2024] GET /v2/images/796d1a83-a325-43af-9c95-f1e098f1467b => generated 957 bytes in 28 msecs (HTTP/1.1 200) 4 headers in 157 bytes (1 switches on core 0)
2024-04-05 10:33:27.792 11 WARNING oslo_policy.policy [None req-20b22860-76c5-4a3a-b592-8ca10b062eed 8000aba2f7aa49b6a8d15a44a39f9eb5 ca09816bc9c148cc8e8f79af1068db97 - - default default] JSON formatted policy_file support is deprecated since Victoria release. You need to use YAML format which will be default in future. You can use oslopolicy-convert-json-to-yaml tool to convert existing JSON-formatted policy file to YAML-formatted in backward compatible way:

could be an issue with the current version of OpenStack (. .. maybe ) many deprecated operations from several versions still work on new versions

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 11:43:28

Yeah I think this is an issue on the OpenStack side and not image-builder. 😞

Kepler SysAdmin
2024-04-05 11:44:22

Do you know what current k8s versions been supported in the image-builder?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 11:45:49

You should be able to override the version in the vars you provide. It should be compatible up to v1.29

👍 Kepler SysAdmin
Kepler SysAdmin
2024-04-05 12:00:33

@Marcus Noble
by the way,
the image start to be upload it:

==> openstack: Waiting for image ubuntu-2204 (image id: 1d34cf8f-ae60-4181-85e0-f0d549d5142c) to become ready...
2024/04/05 12:58:09 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:09 Waiting for image creation status: queued
2024/04/05 12:58:11 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:11 Waiting for image creation status: queued
2024/04/05 12:58:14 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:14 Waiting for image creation status: queued
2024/04/05 12:58:16 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:16 Waiting for image creation status: queued
2024/04/05 12:58:18 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:18 Waiting for image creation status: saving
2024/04/05 12:58:21 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:21 Waiting for image creation status: queued
2024/04/05 12:58:23 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/05 12:58:23 Waiting for image creation status: queued
==> openstack: Error waiting for image: Resource not found
but it does not last, probably the builder goes to the next step too fast

Kepler SysAdmin
2024-04-05 12:16:53

I think Karine is having the same issue:
https://kubernetes.slack.com/archives/C01E0Q35A8J/p1712014119475089

Drew Hudson-Viles
2024-04-08 16:12:22

Just to follow up here as well (been out of the loop for a few days unfortunately) I can confirm I use OpenStack remote on an almost weekly basis and I don't have any issues with it functioning in terms of creating an image at the end of a build. As a result I'd suspect this is an OpenStack config issue rather than an image builder one.

Are you able to use the same OS credentials to manually upload an image using the OpenStack cli?

Kepler SysAdmin
2024-04-08 16:12:52

"Are you able to use the same OS credentials to manually upload an image using the OpenStack cli?"
Yes

Drew Hudson-Viles
2024-04-08 16:13:12

Which version of OpenStack are you running out of interest?

Drew Hudson-Viles
2024-04-08 16:13:27

and how is it deployed - IE Kolla, manually using services etc?

Kepler SysAdmin
2024-04-08 16:14:11

some fresh logs:

==> openstack: Terminating the source server: e68cfe5f-390c-4c5f-b895-861d3f0d47bd ...
2024/04/08 15:12:16 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:16 Waiting for state to become: [DELETED]
2024/04/08 15:12:16 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:16 Waiting for state to become: [DELETED] currently SHUTOFF (0%)
2024/04/08 15:12:19 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:19 [INFO] 404 on ServerStateRefresh, returning DELETED
==> openstack: Creating the image: ubuntu-2204
openstack: Image: 89fdd79b-bf9b-42f8-9b69-c245ce53f945
==> openstack: Waiting for image ubuntu-2204 (image id: 89fdd79b-bf9b-42f8-9b69-c245ce53f945) to become ready...
2024/04/08 15:12:20 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:20 Waiting for image creation status: queued
2024/04/08 15:12:22 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:22 Waiting for image creation status: queued
2024/04/08 15:12:24 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:24 Waiting for image creation status: queued
2024/04/08 15:12:26 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:26 Waiting for image creation status: queued
2024/04/08 15:12:28 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:28 Waiting for image creation status: queued
2024/04/08 15:12:30 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:30 Waiting for image creation status: saving
2024/04/08 15:12:32 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:32 Waiting for image creation status: queued
2024/04/08 15:12:34 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:34 Waiting for image creation status: queued
2024/04/08 15:12:36 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 15:12:36 Waiting for image creation status: queued
==> openstack: Error waiting for image: Resource not found
==> openstack: Provisioning step had errors: Running the cleanup provisioner, if present...
==> openstack: Deleted temporary floating IP '7c1a29c8-1767-4610-8a17-eba3e7cf7c14' (178.73.197.119)
==> openstack: Terminating the source server: e68cfe5f-390c-4c5f-b895-861d3f0d47bd ...
==> openstack: Error terminating server, may still be around: Resource not found
==> openstack: Deleting volume: c5805931-8a3b-42a7-851f-d444d9cd5bef ...
==> openstack: Deleting temporary keypair: packer_66140781-ae67-d1d3-0d7d-9bb69ca0a3c3 ...
2024/04/08 15:12:39 [INFO] (telemetry) ending openstack
==> Wait completed after 8 minutes 6 seconds
2024/04/08 15:12:39 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2024/04/08 15:12:39 machine readable: openstack,error []string{"Error waiting for image: Resource not found"}
==> Builds finished but no artifacts were created.
Build 'openstack' errored after 8 minutes 6 seconds: Error waiting for image: Resource not found

Which version of OpenStack are you running out of interest?
Let me find out

Kepler SysAdmin
2024-04-08 16:14:44

openstack helm charts ( atmosphere from Vexxhost )

Kepler SysAdmin
2024-04-08 16:18:50

"Which version of OpenStack are you running out of interest?"
Bobcat
27.0.1 for glance

Kepler SysAdmin
2024-04-08 16:25:00

if you wonder about glance logs:

# kubectl -n openstack logs pod/glance-api-bd89db7b4-5vjvv | grep 89fdd79b-bf9b-42f8-9b69-c245ce53f945 | grep -i error
Defaulted container "glance-api" out of: glance-api, init (init), glance-perms (init), ceph-keyring-placement (init)
2024-04-08 15:12:30.390 14 ERROR glance_store._drivers.rbd [None req-f045c8b7-f9ea-4ccb-9fea-3baed1a680dc 8000aba2f7aa49b6a8d15a44a39f9eb5 ca09816bc9c148cc8e8f79af1068db97 - - default default] Failed to store image 89fdd79b-bf9b-42f8-9b69-c245ce53f945 Store Exception unable to receive chunked part: OSError: unable to receive chunked part
2024-04-08 15:12:33.695 12 ERROR glance_store._drivers.rbd [None req-f045c8b7-f9ea-4ccb-9fea-3baed1a680dc 8000aba2f7aa49b6a8d15a44a39f9eb5 ca09816bc9c148cc8e8f79af1068db97 - - default default] Failed to store image 89fdd79b-bf9b-42f8-9b69-c245ce53f945 Store Exception unable to receive chunked part: OSError: unable to receive chunked part
2024-04-08 15:12:35.242 9 ERROR glance_store._drivers.rbd [None req-f045c8b7-f9ea-4ccb-9fea-3baed1a680dc 8000aba2f7aa49b6a8d15a44a39f9eb5 ca09816bc9c148cc8e8f79af1068db97 - - default default] Failed to store image 89fdd79b-bf9b-42f8-9b69-c245ce53f945 Store Exception unable to receive chunked part: OSError: unable to receive chunked part

Drew Hudson-Viles
2024-04-08 16:30:21

ok that's a new approach to me so I'm not sure how it's configured out the gate however I know they contributed to my PR which introduced the OpenStack remote approach - I've only got experience with Kolla (and I have infra people who actually spin it up, I just interact with it).

OK, so it works via the CLI - same credential and endpoints etc and it's just the packer approach that's failing - as it has been mentioned this isn't actually image builder itself because by this point it's the OpenStack packer plugin doing the work.

It may be worth reaching out to them in the issues of their helm repo as it could be a simple configuration change that may solve this.

Kepler SysAdmin
2024-04-08 16:31:53

"It may be worth reaching out to them in the issues of their helm repo as it could be a simple configuration change that may solve this"
Who Hashicorp or Vexxhost?

Drew Hudson-Viles
2024-04-08 16:35:20

Vexxhost as it's their chart

Kepler SysAdmin
2024-04-08 16:38:08

Theese are the configs:

  conf:
glance:
DEFAULT:
log_config_append: null
show_image_direct_url: true
show_multiple_locations: true
enable_import_methods: "[]"
workers: 8
cors:
allowed_origins: "**"
image_format:
disk_formats: "qcow2,raw"
oslo_messaging_notifications:
driver: noop
I can change those, let me know

Kepler SysAdmin
2024-04-08 16:40:11

{
"source_image": "",
"networks": "",
"flavor": "",
"floating_ip_network": "public",
"image_name": "ubuntu-2204",
"image_visibility": "public",
"image_disk_format": "raw",
"volume_type": "",
"ssh_username": "ubuntu",
"kubernetes_version": "1.28.7"
}
Is it something to do with the image disk format?

Drew Hudson-Viles
2024-04-08 16:47:46

The image disk format should be fine - this is what I use in my build.

{
"kubernetes_cni_semver": "v1.3.0",
"kubernetes_cni_deb_version": "1.3.0-1.1",
"crictl_version": "1.29.0",
"kubernetes_semver": "v1.29.2",
"kubernetes_series": "v1.29",
"kubernetes_deb_version": "1.29.2-1.1",
"extra_debs": "nfs-common",
"image_name": "",
"source_image": "",
"networks": "",
"flavor": "",
"attach_config_drive": "true",
"use_floating_ip": "true",
"floating_ip_network": "",
"security_groups": "",
"image_visibility": "public",
"image_disk_format": "raw",
"use_blockstorage_volume": "true",
"volume_type": "",
"volume_size": "12",
"qemu_binary": "",
"disk_size": "",
"output_directory": ""
}


As for the configuration I can't advise on that from my side as I've never used Vexxhost's helm charts - I'd recommend having a look at any ingress logs (presuming it deploys them) too as they may reveal some information as to why chunks are failing. The bit that's making me think it's either an OpenStack or at the highest level, a helm chart issue is the OSError: unable to receive chunked part error you're seeing.

👍 Kepler SysAdmin
Kepler SysAdmin
2024-04-08 16:50:40

Thanks

Kepler SysAdmin
2024-04-08 17:09:49

it looks like there is fix by them


trying the above version

GitHub
👍 Drew Hudson-Viles
Kepler SysAdmin
2024-04-08 17:17:53

It looks like is working
2024/04/08 16:17:19 packer-plugin-openstackv1.1.2x5.0linuxamd64 plugin: 2024/04/08 16:17:19 Waiting for image creation status: saving
2024/04/08 16:17:21 packer-plugin-openstackv1.1.2x5.0linuxamd64 plugin: 2024/04/08 16:17:21 Waiting for image creation status: saving
2024/04/08 16:17:23 packer-plugin-openstackv1.1.2x5.0linuxamd64 plugin: 2024/04/08 16:17:23 Waiting for image creation status: saving

Kepler SysAdmin
2024-04-08 17:18:02

It looks like is working:

2024/04/08 16:17:19 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 16:17:19 Waiting for image creation status: saving
2024/04/08 16:17:21 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 16:17:21 Waiting for image creation status: saving
2024/04/08 16:17:23 packer-plugin-openstack_v1.1.2_x5.0_linux_amd64 plugin: 2024/04/08 16:17:23 Waiting for image creation status: saving

Drew Hudson-Viles
2024-04-08 17:18:20

Nice. Fingers crossed! 🤞🤞

Kepler SysAdmin
2024-04-08 17:23:04

It worked!

Drew Hudson-Viles
2024-04-08 17:33:22

Awesome!! Glad it has

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 15:20:28

@mboersma Does CAPZ currently use image-builder to build Flatcar based images and make them available for others to use?

mboersma
2024-04-05 15:56:01

You can build Flatcar images for Azure with image-builder (as you know), but the CAPZ team only publishes reference images for Ubuntu and Windows to the Azure Marketplace.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 15:56:20

Right, nothing I can copy then 😆

I find the whole image gallery / image templates / marketplace images thing in Azure really confusing 😅

👍 mboersma
mboersma
2024-04-05 15:57:47

We want to switch to publishing just to shared image galleries, which would be somewhat simpler (and is also our recommended path for users in our docs). But not there quite yet.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 15:59:21

Yeah, that's what we're doing. 🙂
I'm in the process of switching our source image from being one from another shared gallery to a marketplace image but I was having trouble with it being blocked because billing reasons.
I've finally found the list of approved images which I think solves my problem. 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 16:00:09

I'm just now confused as to why Flatcar has 4 different offers on the marketplace and what the difference is between them (see )

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)
mboersma
2024-04-05 16:07:37

I'm really not sure--the Flatcar team does their own publishing (although not specifically for Cluster API).

I can dig around and try to find out. Also maybe @Mateusz Gozdek (invidian) or @Jeremi Piotrowski knows?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 16:09:01

There's no rush. I've just successfully built and published based on the corevm offer. I just need to test it out now to make sure it actually works 😆
It would be nice to know if there are differences though.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 16:20:04

What's the difference between an Image and a VM image definition?
Image-builder produces both but I'm not sure what is used for what 😕

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 16:50:44

Ah this explains it:
Image is the source that is used to create the version in the image definition 🙂

learn.microsoft.com
Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 16:53:55

I'd like to check some of my understanding with Azure...

With image-builder we create a "managed image" from the VM in our subscription . When the destination is a community gallery this "managed image" is used to create the "version" within the Image definition of our gallery. Is that correct?

If so, do we still need the "managed image" after we've published the version to the gallery? If its safe to delete, is there a way to do this with image-builder?

mboersma
2024-04-05 17:17:23

I think your understanding is correct. As far as I can tell (mostly by having read the docs, not actually experimented), you can create image versions and then delete the managed image since the version is used to provision.

I found this post too that suggests you can do this but it might prevent expanding replication to other regions? Or maybe not. .

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 17:19:03

That's fine, we handle the replication as part of the image-builder run. As long as we don't want extra regions after it's built we're ok. (and we could always just re-build if that was the case)

Now, any idea if image-builder / Packer can automatically clean this up?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 17:20:29

I did come across this issue which seems to suggest that it is meant to be removing the managed image and was broken at one point but then fixed. But I dont see this happening in my environment 😞

GitHub
mboersma
2024-04-05 17:21:09

I'm looking at the docs for the ARM builder and don't see anything that looks relevant to cleaning the managed image.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 17:21:33

Yeah, same 😞

mboersma
2024-04-05 17:22:02

Also in image-builder we're still using the 1.x version of the Azure packer builder plugin, not 2.0 yet.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 17:22:39

Oh. I wonder if that could be part of it 🤔

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 17:22:57
This pull request first appeared in v1.7.1
mboersma
2024-04-05 17:23:02

Because we still need support for VHD publishing with SAS URLs which I think 2.x removes.

mboersma
2024-04-05 17:26:18

The plugin is up to 2.0.5 now:
Upgrading requires CAPZ to move to community gallery publishing IMHO, which is something I'm working on but not there yet.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-05 17:27:32

Ok. It's not a huge problem. I was just trying to figure out what was going on in our account. I'm not too bothered with the Images being left behind. Maybe when we finally upgrade they'll magically start being cleaned up and I don't need to think about it 😄

mboersma
2024-04-05 17:27:59

One can hope!

Slackbot
2024-04-08 15:30:09

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-08 17:05:19

⚠️ Currently all PRs are failing due to an issue with photon-4 packages not matching known keys. This is causing the pull-ova-all tests to fail for everyone.

If anyone happens to know what might have happened to cause this and how to fix it that's would be a huge help. If we don't track down what the problem is in the next couple days we'll remove the photon-4 OS from the test to unblock PRs while we figure out the problem. (Note, this doesn't seem to effect photon 3 or 5)

Issue:

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-11 13:17:51

I'd like to better understand how Packer make use of SSH keys when building image with image-builder and I have a couple questions I'm not clear on:

  1. What is this SSH key? It has the associated user as capi but the name property in the users list is capv 😕 If I'm following correctly this is used in image-post-create-config.sh which is then used by the OVA provider which makes the capv make sense. Not sure if those files in cloudinit are used by any other providers though. Can we remove this ssh key?

  2. Is there any reason for us not to add sshclearauthorizedkeys (docs) to image-builder so that the ~/.ssh/authorizedkeys file is emptied before we publish the images?

Paramita
2024-04-15 13:06:34

@Paramita has joined the channel

Anubhav Gain
2024-04-16 04:44:31

@Anubhav Gain has joined the channel

Abhay Krishna Arunachalam
2024-04-17 08:17:44

Hi image-builder maintainers! I wanted to know if there's a tentative date for the next release of image-builder v0.1.26.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-17 08:25:33

I want to get https://github.com/kubernetes-sigs/image-builder/pull/1438 merged in first now that it’s unblocked by the failing test then I think we should be good to go.

GitHub
Abhay Krishna Arunachalam
2024-04-17 08:29:16

Nice, thanks! yay-fox

xinity
2024-04-19 11:59:31

@xinity has joined the channel

xinity
2024-04-19 12:03:27

hello there, back into capi world i'm trying to build flatcar qemu image from the repository but failing miserably 😞 the latest log i'm having is:

2024/04/19 10:52:11 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:11 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 [INFO] Attempting SSH connection to 127.0.0.1:2686...
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 [DEBUG] reconnecting to TCP connection for SSH
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 [DEBUG] handshaking with SSH
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 Keyboard interactive challenge:
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 -- User:
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 -- Instructions:
2024/04/19 10:52:18 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:18 -- Question 1: Password:
2024/04/19 10:52:20 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/04/19 10:52:20 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey keyboard-interactive], no supported methods remain
tested with plain ssh connection and same behavior 😞

anyone may have a clue to fix this ?

Jeremi Piotrowski
2024-04-19 13:59:29

what is your host ssh version?

xinity
2024-04-19 14:05:19

8.7p1-34.el9.x86_64

xinity
2024-04-19 14:05:33

i'm trying to build on rockylinux9

Jeremi Piotrowski
2024-04-19 14:10:00

thats a very old ssh version and you may have protocol issues because of that

Jeremi Piotrowski
2024-04-19 14:10:05

check this commit: 74f31d3b4b5b7fb12fd3340f6fb35d9cfdc26d67

Jeremi Piotrowski
2024-04-19 14:10:25

or i suggest running the build from a docker container with a more recent ssh client

xinity
2024-04-19 14:15:40

how would you build from a docker container ? that's an interesting point of view

Jeremi Piotrowski
2024-04-19 14:16:37

there is a Dockerfile in images/capi

Jeremi Piotrowski
2024-04-19 14:16:50

you can use that as a build environment

Jeremi Piotrowski
2024-04-19 14:17:12

if you need qemu then run privileged or pass through /dev/kvm

❤️ xinity
xinity
2024-04-19 14:25:15

will try with the docker build mechanism

xinity
2024-04-19 14:25:17

will keep you posted

xinity
2024-04-19 20:53:44

tried finally with the docker image, same issue @Jeremi Piotrowski 😢

xinity
2024-04-20 12:22:44

i've opened a bug report :

GitHub
:thank_you_icon: Marcus Noble
Mike Tritabaugh
2024-04-19 21:39:31

@Mike Tritabaugh has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2024-04-22 11:52:51

There's currently nothing on the agenda for todays image-builder office hours so unless someone has anything they'd like to bring up I suggest we skip for this week.

I do think it's time for us to do a new release though now that we're unblocked by the failing tests so if anyone has any PRs they'd like to get merged in before we do the next release please speak up now 🙂

mboersma
2024-04-22 15:24:59

+1 from me on both points.

👍 Marcus Noble
Slackbot
2024-04-22 15:30:04

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma
2024-04-22 16:32:44

No topics on the list, so we skipped today. See you on May 6th!

Kishore
2024-04-30 07:25:16

@Kishore has joined the channel

Kishore
2024-04-30 07:40:00

Hi,

I am creating windows image for OCI. Below is the PACKERVARFILES specification. But not able to build Windows image. Is there anything missed in this configuration.



{
"buildname": "windows",
"base
imageocid": "ocid1.image.oc1.eu-frankfurt-1.aaaaaaaaiwwre36icxfiivmgqlfdbrjm67igscbikjq4k2luhbjgcwyxiywa",
"ocpus": "128",
"shape": "BM.Standard.E4.128",
"region": "eu-frankfurt-1",
"compartment
ocid": "ocid1.compartment...",
"subnetocid": "ocid1.subnet.oc1..",
"availability
domain": "DDJb:EU-FRANKFURT-1-AD-1",
"userocid": "ocid1.user..",
"fingerprint": "af:66:ce:6b:63:d1:ef:99:97:43:50:36:35:f2:71:f9",
"tenancy
ocid": "ocid1.tenancy....",
"keyfile": "~/.oci/ociapikey.pem"
}
/usr/bin/packer build -var-file="/home/ubuntu/image-builder/images/capi/packer/config/kubernetes.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/windows/kubernetes.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/containerd.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/windows/containerd.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/windows/ansible-args-windows.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/common.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/windows/common.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/windows/cloudbase-init.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/goss-args.json" -var-file="/home/ubuntu/image-builder/images/capi/packer/config/additional
components.json" -color=true -var-file="/home/ubuntu/image-builder/images/capi/packer/oci/windows-2022.json" -var-file="/home/ubuntu/image-builder/images/capi/oci.json" packer/oci/packer-windows.json
Error: Failed to prepare build: "oracle-oci"

3 error(s) occurred:

* 'fingerprint' must be specified
* 'securitytokenfile' must be correctly specified. did not find a proper
configuration for key id
** 'key_file' must be correctly specified. did not find a proper configuration
for private key

Kishore
2024-04-30 09:04:06

Got resolved. Added missing items in packer-windows.json

🎉 Marcus Noble, mboersma, Drew Hudson-Viles
georgeb
2024-04-30 12:41:42

@georgeb has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-05 19:28:26

It’s a public holiday in the UK tomorrow so I’ll be skipping the office hours. There’s currently nothing on the agenda so feel free to skip if nothing comes up. 🙂

☝️ Drew Hudson-Viles, mboersma
Slackbot
2024-05-06 15:30:15

Reminder: Image-Builder office hours start in 1 hour. Agenda:

xinity
2024-05-10 09:03:29

any idea why when i built a flatcar image , the ssh pub key isn't populated when used with CAPI ?

Kepler SysAdmin
2024-05-14 13:33:20

Hello All,
the image-builder works wonderfully,
Now, I know this is just for build images with pre-built k8s.
Is there something similar but with pre-built wordpress?

voor
2024-05-14 13:35:05

You would deploy a kubernetes cluster and then run WordPress. https://bitnami.com/stack/wordpress

bitnami.com
Kepler SysAdmin
2024-05-14 13:38:45

Hi....
yes, but i need this as a openstack image
already have it as on a k8s cluster

voor
2024-05-14 16:02:43

If you need an openstack image you could look at

Or you could use packer directly for openstack

Medium
Reading time
3 min read
developer.hashicorp.com
syt
2024-05-17 02:49:09

@syt has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 09:06:03

I was planning to cut a new release today as it's been a while since the last and we've had quite a few PRs merged since then. Any PRs that people would like to try and get in before I do?

Drew Hudson-Viles
2024-05-20 09:12:24

I'm good from my side, thanks!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 13:19:43

ok, I'm going to start a new release now.

👍 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 13:37:12

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 13:50:59

Docs PR:

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 13:55:46

@Drew Hudson-Viles

tide Pending Not mergeable. Needs approved, lgtm labels.

Drew Hudson-Viles
2024-05-20 13:56:19

Sorry, did a silly and forgot the comment when approving 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 13:56:33

It's ok. I find it weird that lgtm is still needed with approve 😆

☝️ Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 14:00:20

laugh-cry

tide Pending Not mergeable. Needs approved label.

Drew Hudson-Viles
2024-05-20 14:01:04

Damn you automation! 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 14:01:18

shake_fist

Drew Hudson-Viles
2024-05-20 14:01:27

So even though I approved, I needed to do /approve... nice, nice. 😄

Drew Hudson-Viles
2024-05-20 14:01:46

All done buddy

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 14:02:04

yeah, tide doesn't use the GitHub PR status. It keeps its own state based on comments / labels

Drew Hudson-Viles
2024-05-20 14:02:13

That's fair 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 14:02:21

It... frustrating sometimes 😆

😆 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 13:52:04

Image-builder v0.1.26 is now available:
Thanks to all contributors! 🎉

GitHub
🎉 Drew Hudson-Viles, Abhay Krishna Arunachalam, jsturtevant, Mitchel Haring
Abhay Krishna Arunachalam
2024-05-21 05:36:21

Nice!

Slackbot
2024-05-20 15:30:26

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 15:48:49

@mboersma do you still want to talk about Ubuntu 24.04?

mboersma
2024-05-20 15:57:14

Sure, we can move that to today, but it's a short topic.

👍 Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-20 16:35:07

@Drew Hudson-Viles are you joining? You have an item on the agenda

Drew Hudson-Viles
2024-05-20 16:35:25

I am, I'll be 2 minutes, just on a call.

Sriraman Srinivasan
2024-05-21 04:23:43

Do we have a recording of this?

Sriraman Srinivasan
2024-05-21 04:28:03

Sorry found the link in the doc

👍 mboersma
Sriraman Srinivasan
2024-05-21 04:25:49

Just wondering about the impact of this in IB

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-23 15:29:13

@mboersma I think this change has broken Flatcar on AWS 😞

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-23 15:29:28
    amazon-ebs.flatcar-stable: TASK [providers : Install AWS CLI v2] ************************************************************************************
amazon-ebs.flatcar-stable: fatal: [default]: FAILED! => {"changed": true, "cmd": ["/tmp/aws/install", "-i", "/usr/local/aws-cli", "-b", "/usr/local/sbin"], "delta": "0:00:01.552103", "end": "2024-05-23 14:19:56.504588", "msg": "non-zero return code", "rc": 1, "start": "2024-05-23 14:19:54.952485", "stderr": "mkdir: cannot create directory '/usr/local/aws-cli': Read-only file system", "stderr_lines": ["mkdir: cannot create directory '/usr/local/aws-cli': Read-only file system"], "stdout": "", "stdout_lines": []}
amazon-ebs.flatcar-stable:
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-23 15:29:49

What was the reason for removing these aws cli changes specifically?

mboersma
2024-05-23 15:34:44

Oops! It was to support Ubuntu 24.04 on AWS, I unified the couple of paths that used to be there to install the AWS CLI (and added gpg checksum validation). I didn't realize that would pull in Flatcar actually.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-23 15:36:09

I'm going to try and get a PR up.

mboersma
2024-05-23 15:37:27

Thanks Marcus! LMK if you need help or want me to fix my own damage, I'm just waking up but I'd be glad to look into it.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-23 15:37:50

I think I have it already. Haven't tested it yet but I'll get the PR up for you to take a look at

👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-23 15:39:36

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-23 16:04:25

Confirmed working for Flatcar

👍 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-24 07:09:18

I'm going to get a new release out with this fix today

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-24 07:51:46

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-24 08:14:13

And then once done I have this for the docs:

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-24 09:15:50

@Drew Hudson-Viles as you're about would you mind looking at the above? 🙏 🥺

Drew Hudson-Viles
2024-05-24 09:16:16

Sure can, hold please!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-24 09:16:28

My hero! hero

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-24 09:19:34

(and the one above it 😉 )

Drew Hudson-Viles
2024-05-24 09:19:42

yeah I'm on it now 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-24 09:19:49

💙

Drew Hudson-Viles
2024-05-24 09:20:19

all done 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-24 09:21:01

Thank you kindly sir 🙂

:gr_captain_salute: Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-24 09:31:01

Image-builder v0.1.27 is now available:

This is a small release with just a couple bug fixes in it. 🙂

Thanks to all contributors!

GitHub
:emoji_k8s_loft: Drew Hudson-Viles, jsturtevant, Abhay Krishna Arunachalam, Mitchel Haring, mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-27 18:32:34

Image-builder v0.1.28 is now available:
Thanks to all contributors!
(Trying to do more frequent, smaller releases. Lets see how this goes 😄)

GitHub
🎉 Drew Hudson-Viles, Mitchel Haring
Jacob Weinstock
2024-05-28 01:16:01

@Jacob Weinstock has joined the channel

Jonathan Rosser
2024-05-28 15:11:03

@Jonathan Rosser has joined the channel

Sriraman Srinivasan
2024-05-29 07:46:02

With PR for license exception on packer getting rejected, whats the plan going forward?

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-29 07:47:47

It's unclear. Please see the discussion in our tracking issue:

For right now nothing changes as we're still pinned to the pre-BUSL version but that can't continue forever.

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-05-29 07:48:54

I have also added it to the agenda for next weeks office hours though I'm not sure there will be any solution come out of that. It'll likely be more to make the current state of things clear.

Jonathan Rosser
2024-05-29 14:40:46

i think there is another file to clean up on ubuntu images etc/cloud/cloud.cfg.d/90-installer-network.cfg - see

GitHub
Jonathan Rosser
2024-05-29 14:57:23

i found this by trying to use the 22.04 daily build iso instead of the 22.04.04 release ISO, and you get revision 5495 vs 5741 of the subiquity snap in those different ISOs

Kumar
2024-05-29 22:47:21

@Kumar has joined the channel

Christopher
2024-06-01 09:01:02

@Christopher has joined the channel

Slackbot
2024-06-03 15:30:13

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-06-04 10:10:10

Just catching up on the office hours... some notes:

  • I spoke to the Flatcar people at KubeCon who are trying to get sysext support added to Ubuntu and other OSs too

  • I'm happy with Windows being a different approach to others, that's an acceptable split to me

  • At Giant Swarm we make heavy use of the image-builder container image so we'd need to switch to building our own image (we actually already do with the upstream image-builder one as the base)

  • If sysext is the way forward for Linux OSs then I'm not sure there's a need for image-builder going forward (which is fine if that is the direction things are going)

  • I agree that chasing up CNCF again and clarifying our usage is worth doing. I did ask on the issue we already had but haven't yet had a reply from the CNCF member that closed it. (I also don't think I agree with Fabrizio that Packer isn't a runtime dependency for us. The image we produce isn't "image-builder" the process of building is image-builder so that is the runtime in my opinion - but I'm not a lawyer)

  • Yeah agree it'd be good to have Flatcar on the call at some point. I suspect that @Thilo Fromm might be willing to chat with us as we had a really good chat about it at KubeCon.

  • If using sysext then it should be possible to use this rather than image-builder to produce the images -

  • bootc only supporting Fedora/CentOS is too limited right now in my opinion.


/cc @Drew Hudson-Viles @mboersma FYI 🙂

👍 Drew Hudson-Viles, mboersma, Lennart Jern, jawnsy
👍:skin_tone_2: Anurag
Sriraman Srinivasan
2024-06-06 05:45:56

My couple of cents:

I also don't think I agree with Fabrizio that Packer isn't a runtime dependency for us. The image we produce isn't "image-builder" the process of building is image-builder so that is the runtime in my opinion - but I'm not a lawyer
This is my understanding. Please correct me if I am wrong here. The output of the whole process (image-building) are artifacts which are later consumed by CAPX to bootstrap K8s cluster. We use packer to bootstrap VMs in providers and configure further changes and then produce the final artifacts (in case of vSphere OVA, AWS AMI, etc). The use of packer here is only for building the artifacts (or the final desired output) and not while actual cluster creation(this is what I assume is actual runtime - cluster creation). So not sure why calling packer a build time dependency or usage for producing the images is improper.
Again please excuse/correct me if I am wrong here.

Regarding the other points +1. And sorry wanted to attend the SIG call but was out sick, so could not make it.

👍 Lennart Jern
mboersma
2024-06-06 16:39:19

The use of packer here is only for building the artifacts
I agree with this, and my reading of the BUSL license is that's all they care about as well. Hopefully we can get the CNCF to agree.

Thilo Fromm
2024-06-07 09:15:24

OH NO I missed this - was on an off-site all week. I am absolutely available for chatting on CAPI sysexts! @tormath1 is also a good peer for this discussion.

💙 Marcus Noble, tormath1, Anurag, Sriraman Srinivasan
:_thank_you_: Anurag, mboersma
Thilo Fromm
2024-06-07 09:16:49

From all the CAPI providers I guess the OpenStack folks (CAPO) are most experienced with using sysexts at this point. They use Flatcar + Kubernetes sysext a lot for CI / Testing etc. and they really like it.

Mark Kamsika
2024-06-04 10:55:22

@Mark Kamsika has joined the channel

rajas
2024-06-05 06:30:25

@rajas has joined the channel

Anurag
2024-06-07 20:02:46

Hello, have discussed using instead of pip3 as of now in the image-builder repository? Would you be open to it in the current state of the project?

context: I develop mostly from a BM server and I think this would be faster so just initiating a discussion about the same.

mboersma
2024-06-12 19:31:08

That would be interesting to see. uv does seem like a superior tool.
It's mostly a matter of supporting multiple distros cleanly (and Windows). Ideally we stick with "in-the-box" tools packaged by the distro vendor.
We haven't had many actual resolver problems with pip recently, but it's been tricky to install extra packages now that distros want you to use --break-system-packages and --user doesn't always work. Maybe uv would be more flexible here?

Karine Santos
2024-06-11 21:00:22

Hello everyone!
Is it possible to update the kernel version with image builder?

mboersma
2024-06-12 19:25:43

Not directly. image-builder starts from a source image that includes the kernel and all the basic tools, so for the kernel you'll get whatever comes with Ubuntu 24.04 (for example) as packaged for your cloud provider.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-06-13 14:43:02

Image-builder v0.1.30 is now available:
Thanks to all contributors! 🎉

🚀 mboersma, Sriraman Srinivasan, rajas, Anurag
eSean
2024-06-13 15:00:51

@eSean has joined the channel

Jonathan Rosser
2024-06-17 13:23:21

could i get some pointers on the right way to fix the broken azure tests? I can make the whole grub update be conditional on update-grub being installed?

GitHub
Slackbot
2024-06-17 15:30:19

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-06-17 15:33:52

As the agenda is currently empty and at least 2 of the maintainers are unable to make it today we're going to skip the office hours this week.

✅ Anurag
Kevin Breit
2024-06-19 02:37:44

@Kevin Breit has left the channel

mloza
2024-06-21 21:22:46

@mloza has joined the channel

Jasper
2024-06-26 14:09:41

@Jasper has left the channel

dalees
2024-06-27 06:08:30

@dalees has joined the channel

bharath2438
2024-07-01 06:07:19

@bharath2438 has joined the channel

Slackbot
2024-07-01 15:30:29

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma
2024-07-01 16:05:30

We don't have any topics in the agenda currently, so let's skip office hours this week. Please reach out on the Slack channel if you have any questions or need support with image-builder.

Justinas B
2024-07-08 09:22:03

@Justinas B has joined the channel

Justinas B
2024-07-08 10:06:24

Hey! I am facing an issue when building an image - the VM gets stuck on language selection screen and it seems cloud-init scripts are not being executed. Any ideas how to fix that?

Jeremi Piotrowski
2024-07-08 10:27:56
export DEBIAN_FRONTEND=noninteractive
Jeremi Piotrowski
2024-07-08 10:28:05

before running apt-get steps

Justinas B
2024-07-08 11:02:03

I dont see apt-get being invoked anywhere on proxmox build ()

GitHub
Justinas B
2024-07-08 11:02:15

Right after below i get language selection screen

Justinas B
2024-07-08 11:02:26

any ideas where i should put that export DEBIAN_FRONTEND=noninteractive?

Justinas B
2024-07-08 11:04:50

In logs i see following error "GET /api2/json/nodes/pve-01/qemu/118/agent/network-get-interfaces HTTP/1.1" 500 13, which i assume is caused by qemu agent not running on machine

Justinas B
2024-07-08 13:02:17

Could someone please confirm that image-builder works only if there is a DHCP server to assign IP addresses automatically in proxmox case? It seems that in my case build process gets stuck due to connectivity issues as VM never gets IP address assigned

Nicolò Ciraci
2024-07-08 13:24:45

Hi! Yes, we use it in the same scenario. No issues at all with a simple image with basic Ubuntu. Given the fact that this images should be used primary with CAPI I assume DHCP is always needed. Otherwise you would need to use CloudInit to spin up VMs with already assigned IPs.

Justinas B
2024-07-08 14:19:46

ok, thanks @Nicolò Ciraci! I have spinned up new DHCP server and i see it offered IP address to ubuntu-server

Justinas B
2024-07-08 14:20:29

but my ubuntu server seems to be stuck here

Justinas B
2024-07-08 14:21:14

sorry for dumb question - but any ideas why this could be happening and how to debug that?

Justinas B
2024-07-08 14:21:24

packer logs shows below:

2024/07/08 16:15:05 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:05 [INFO] Waiting 5s
2024/07/08 16:15:11 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:11 [INFO] Waiting 5s
2024/07/08 16:15:16 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:16 [INFO] Waiting 5s
2024/07/08 16:15:24 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:24 [DEBUG] Unable to get address during connection step: 500 Internal Server Error
2024/07/08 16:15:24 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:24 [INFO] Waiting for SSH, up to timeout: 2h0m0s
==> proxmox-iso.ubuntu-2204: Waiting for SSH to become available...
2024/07/08 16:15:27 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:27 [DEBUG] Error getting SSH address: 500 Internal Server Error
2024/07/08 16:15:35 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:35 [DEBUG] Error getting SSH address: 500 Internal Server Error
2024/07/08 16:15:43 packer-plugin-proxmox_v1.1.8_x5.0_darwin_arm64 plugin: 2024/07/08 16:15:43 [DEBUG] Error getting SSH address: 500 Internal Server Error

Nicolò Ciraci
2024-07-08 14:34:03

The packer errors are caused by the fact that your instance of Ubuntu is stuck, but I don’t see any clear issue in the logs. Are DNS reachable?

Justinas B
2024-07-08 14:45:43

yeah, if I restart machine and and login through serial console, i am able to manually deploy qemu-guest-agent

Justinas B
2024-07-08 14:46:03

but it needs restart, as it is literraly stuck

Justinas B
2024-07-08 14:46:10

i do not see any network activity happening on it

Nicolò Ciraci
2024-07-08 15:03:49

The image-builder stack has some additional flag by which you can install additional package. Something like this PACKERFLAGS="--var 'extradebs=\"qemu-guest-agent\"'" make build-qemu-ubuntu-2204-crio ; you’ll need to adjust the command based on you linux flavour.

Justinas B
2024-07-10 07:02:18

OK, so it seems the issue was that my laptop where i ran packer was not accessible from the temporary VM

Justinas B
2024-07-10 07:02:30

i have moved the build server to proxmox itself and it worked fine

chrischdi
2024-07-11 16:58:39

@chrischdi has joined the channel

chrischdi
2024-07-11 16:59:33

Hey folks, I opened an issue regarding the mandatory pull-ova-all due to the upcoming changes by test-infra / to the prow CI, PTAL:

GitHub
Slackbot
2024-07-15 15:30:26

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-07-15 15:33:01

Agenda is currently empty. Does anyone have any topics they'd like to discuss or shall we skip?

mboersma
2024-07-15 15:39:38

Nothing in particular on my end, happy to skip it or to carry on if someone has a discussion topic.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-07-15 16:03:45

ok, unless someone shouts up in the next 30 min I'm going to skip. Could do with a rest after work to be honest 🙂

❤️ mboersma
Drew Hudson-Viles
2024-07-15 16:11:02

What about the OVA removal that's happening? Worth chatting abotu that or shall we put it off until next time? I've got nothing to add just wasn't sure if a discussion was needed 😄

Drew Hudson-Viles
2024-07-15 16:11:32

But yeah. I'm running on 3 hours sleep thanks to poorly child so I would appreciate not going on YouTube this week 😄

mboersma
2024-07-15 16:16:57

I don't have much insight into the OVA changes, but it would be good to hear from someone more involved. If we're not up for it this week, maybe we can recruit someone to summarize next time.

Drew Hudson-Viles
2024-07-15 16:17:33

yeah that sounds like a good plan

Marcus Noble (k8s@marcusnoble.co.uk)
2024-07-15 16:18:10

Yeah agreed

Karine Santos
2024-07-17 13:55:49

Hello Everyone!
I'm trying to upload an image from 1.29 and 1.30 of k8s, and I saw that the cni-plugin was upgraded from that version (before I used v1.2.0 of the cni-plugin, and from version 1.29 and 1.30 of k8s , cni-plugin updated to v1.3.0 and v1.4.0), but I get the following error when trying to update cni-plugin configuration in packer?

Karine Santos
2024-07-17 13:56:24

Before I configured it this way. Here are the changes I made:

Karine Santos
2024-07-17 13:56:54

The error I get:

openstack: fatal: [default]: FAILED! => {"cache_update_time": 1721186207, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'kubelet=1.29.6-1.1' 'kubeadm=1.29.6-1.1' 'kubectl=1.29.6-1.1' 'kubernetes-cni='' failed: E: Version '' for 'kubernetes-cni' was not found\n", "rc": 100, "stderr": "E: Version '' for 'kubernetes-cni' was not found\n", "stderr_lines": ["E: Version '' for 'kubernetes-cni' was not found"], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nPackage kubernetes-cni is not available, but is referred to by another package.\nThis may mean that the package is missing, has been obsoleted, or\nis only available from another source\n\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "Package kubernetes-cni is not available, but is referred to by another package.", "This may mean that the package is missing, has been obsoleted, or", "is only available from another source", ""]}
4116 openstack:
4117 openstack: PLAY RECAP **
4118 openstack: default : ok=44 changed=34 unreachable=0 failed=1 skipped=190 rescued=0 ignored=0
4119 openstack:

Anurag
2024-07-17 17:03:30

I think you'll have to set 1.4.0 for kubernetescnideb_version at least given you're using apt.

Is there a reason to set to it to null? Looking at the config I think that should be 1.4.0

Sriraman Srinivasan
2024-07-18 13:27:43

Interesting discussion -->

GitHub
Junfei Zhang
2024-07-23 00:50:37

@Junfei Zhang has joined the channel

Junfei Zhang
2024-07-23 01:10:13

Hey everyone! quick question is this PR still functional ? In other word, is building arm64 image on mac m1 supported by image-builder?

GitHub
:think3d: Phil H
Vladu
2024-07-23 09:01:58

Hello, I am the creator of the PR. This PR was done aa year ago and will probably not work anymore because of the now inexistent links to the k8s binaries. Still, with some changes to the urls, it might work on a native arm64 Linux box. If I remember correctly, I know Mac M1 does not actually support kvm qemu-arm64? if that is the case, you need to use a Linux arm64 box to try out the PR.

Vladu
2024-07-23 09:03:13

We can continue this discussion on the PR github thread, to let me know if you are constrained to running this on Mac M1. Fortunately, I also own a Mac M1 and I can check it out too.

Marius Oprin
2024-07-23 04:19:58

@Marius Oprin has joined the channel

Jay J
2024-07-23 05:56:21

@Jay J has joined the channel

Vladu
2024-07-23 08:59:19

@Vladu has joined the channel

feitnomore
2024-07-25 00:32:00

does the images for kubevirt/qemu support EFI/Secure Boot? I've tried to boot a few of them and failed

feitnomore
2024-07-25 00:35:18

follow up question: is it possible to create an image that supports both, bios and efi, as well as secureboot? I see cloud images like ubuntu support that

Jeremy Maes
2024-07-25 11:10:52

@Jeremy Maes has joined the channel

Tim Cheng
2024-07-26 05:06:45

@Tim Cheng has joined the channel

Slackbot
2024-07-29 15:30:22

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Sriraman Srinivasan
2024-07-30 11:30:25

@Marcus Noble /@mboersma Need help with merging PR. This is regarding Ubuntu-24.04 support for vSphere environment. All the CI jobs are successful. It would be great if we can get this in before end of this month (next two days) - disabling of pre-submit CI for all OVA issues
cc: @chrischdi @rajas

GitHub
Christian Schlotter (https://kubernetes.slack.com/team/UBVL9Q4TB)
:this: rajas
🎉 chrischdi, rajas
:lgtm: mboersma
rajas
2024-07-30 11:52:07

Yay! Finally the CI is happy! 🎉

Sriraman Srinivasan
2024-07-31 05:52:24

Thanks @mboersma @Drew Hudson-Viles for the approval and merge in time...

:gr_captain_salute: Drew Hudson-Viles
:thank_you: mboersma
bavarianbidi
2024-07-30 14:28:38

Hey folks,
question-block regarding the build process how it's implemented in image-builder.

not really related to image-builder project itself - but i see a large user base here --> more distributed knowledge 🤞

What's the main reason why image-builder doesn't build a generic raw and convert it later into a qcow2 (openstack-square ), an ami (aws ) or vhdx (azuretent )?

just asking because:

  • i had to build a generic OS image which has to be used on dedicated cloud platforms

  • and TIL thatqemu-img is able to convert images


Does it brings any downside for using a converted image, compared to a dedicated build artifact for e.g. Azure)? 🧵

bavarianbidi
2024-07-30 14:29:40

Or is it for historical reasons, that no generic build was implemented as in the beginning of image-builderit was easier to build on the target cloud-providers directly?

chrischdi
2024-07-30 16:04:34

I think one point is e.g.: For azure you may want to have different config for e.g. cloud-init or tools installed e.g. the azure cli, compared to when running on openstack.

There are some ansible tasks which are run depending on the target which gets built.

bavarianbidi
2024-07-31 06:18:15

thank you Christian, yes that's a valid point.
And if i take a look at the provider specific ansible tasks, they seems to be valid for an image-builder perspective (like you said, having azure-cli in there or the ssm agent for AWS.

but if you do not really need these, do you know any limitiation when it came to conversion via qemu-img convert

chrischdi
2024-07-31 07:37:34

Nope, not really. If the image still has everything you need, I don’t see a reason why it should not work, except maybe hypervisor specific things at the end, but if it runs it runs 🙂

:thx_thanks: bavarianbidi
chrischdi
2024-07-31 07:38:00

Note: I’m not very familiar with aws and azure image building.

:this_is_fine_fire: bavarianbidi
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-01 10:24:13

🤔 I wonder if this is a good approach for us to try out switching to using systemd sysext. Instead of trying to update all existing targets we create a new one specifically for sysext that builds a raw image with optional conversion at the end for each cloud provider. That convert stage could also possibly layer on provider-specific sysext packages like azure-cli.

chrischdi
2024-08-01 10:50:40

That would be way more efficient ack

tormath1
2024-08-01 10:54:49

Giving my 2 cents, this is how Flatcar is actually built: there is a generic image build then it's converted to cloud providers VM images with specific bits (image format, OEM tools provided as sysext, etc.)

👍 Marcus Noble
:thx_thanks: bavarianbidi
mboersma
2024-07-31 00:06:42

Image-builder v0.1.31 is now available: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.31
Thanks to all contributors! 🎉

:awesome_go: chrischdi, Anurag
:emoji_k8s_loft: Drew Hudson-Viles, Anurag, jsturtevant
hbarel
2024-08-01 13:27:17

@hbarel has joined the channel

Erkan Erol
2024-08-02 17:08:52

@Erkan Erol has left the channel

quba
2024-08-05 16:32:21

@quba has left the channel

bentheelder
2024-08-06 23:53:11

@bentheelder has joined the channel

bentheelder
2024-08-06 23:53:53

PTAL, there's a bug in acquiring GCP projects that is causing leaked CI projects

GitHub
:done2: jsturtevant
dims
2024-08-07 01:34:23

@jsturtevant around? PTAL if so!

dims
2024-08-07 01:47:59

thanks a ton @jsturtevant

💯 bentheelder
❤️ bentheelder
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-09 10:08:09

Has anyone been able to successfully build the latest 3975.2.0 release of flatcar Flatcar for CAPV using make build-node-ova-vsphere-flatcar?
I'm getting stuck at the "Waiting for IP..." stage but currently don't have access to vcenter to debug the issue 😞
(The exact same image-builder setup works fine with Flatcar 3815.2.5 so I suspect something has changed in the new release that we need to handle)

chrischdi
2024-08-09 10:27:49

what could help for debugging (also in future) is try to get a screenshot of the vm to the artifacts using govc vm.console -capture screen.png my-vm

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-09 10:28:26

I don't currently have access to vcenter. Needing another team to take a look for me.

chrischdi
2024-08-09 10:28:50

I’m planning to build an image next week for CAPV with v1.31 when its released.

chrischdi
2024-08-09 10:29:31

I don’t currently have access to vcenter. Needing another team to take a look for me.
I was thinking of having this command somewhere in the image-builder pipeline for failures 🙂 to always get this in case of an error, also later when ci on vsphere is back.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-09 10:29:45

Oh interesting idea!

chrischdi
2024-08-09 10:30:49

or at least being able to toglge it on manually on a PR or so if it can’t be always on 🙂

:nod: Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-09 10:31:36

But thinking about it, in my CI pipeline I don't actually have any way currently to save artifacts like that so would need a bit of a rewrite on my end anyway

chrischdi
2024-08-09 10:32:05

ah ok 😞 fair point

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-09 10:32:29

Still, extra debugging of failures is always helpful

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-09 16:09:34

Update: Looks like the latest Flatcar stable release has a considerable slower boot time (by 1m30s) that is causing problems with the default bootwait time we have set in image-builder not being long enough.
To work around this issue until Flatcar has worked out what is the cause of the slowdown you can set the boot
wait Packer variable to something like 120s in your user provided vars.

:thx_thanks: chrischdi
chrischdi
2024-08-13 19:22:27

120s was not enough over here, did use 180s instead (~145s would have been good enough).

Most time is for:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-13 19:24:22

Yeah, exactly that. I found 120s worked for me ( and I have a PR to set that) but it doesn’t hurt to have a little more possibly.
I did also have trouble with the boot command itself failing too.

👍 chrischdi
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-13 19:24:37

https://github.com/kubernetes-sigs/image-builder/pull/1540

GitHub
chrischdi
2024-08-13 19:25:46

Let’s see if that commands worked for me 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-13 19:45:06

I don’t understand why it’s failing in the latest flatcar release though

chrischdi
2024-08-13 19:46:01

which one?

With your PR I’m now at the ansible stuff (stable channel / 3975.2.0)

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-13 19:52:34

I mean I don’t understand what changed in the latest flatcar that has caused what was there before in image builder to fail

:nod: chrischdi
chrischdi
2024-08-13 19:52:56

If I find time I may take a look the next days.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-13 20:00:12

There’s some updates from the flatcar team in that issue I opened

Slackbot
2024-08-12 15:30:24

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-12 15:39:46

☝️ Agenda is currently empty. Anyone got anything they'd like to discuss or shall we skip?

mboersma
2024-08-12 16:05:13

I don't have anything in particular, happy to skip unless someone adds something to the agenda.

👍 Marcus Noble
Drew Hudson-Viles
2024-08-12 16:33:34

Sorry for the late reply, yes I'm happy to skip. I'm actually helping my brother move today anyway.

Dhilip Shankaranarayanan
2024-08-12 16:00:41

@Dhilip Shankaranarayanan has joined the channel

bentheelder
2024-08-12 16:52:20

@bentheelder has left the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-13 10:19:56

Can someone please help me sanity check if an assumption I previously had about AMI builds is actually incorrect?

In my image-builder pipelines I'm setting the FLATCARVERSION env var to specify what Flatcar image to base my build on. This works as expected on Azure thanks to setting the distributionversion in the packer vars but I don't think this env var is used when building AWS AMIs. 😞

For AMIs we use the following AMI filter:

Flatcar{{env FLATCAR&#95;CHANNEL}}
which just seems to take into consideration the flatcar channel.

For reference, a Flatcar AMI name looks something like this: Flatcar-stable-3975.2.0-hvm

Unless I'm mistaken, this means that when building AMIs image-builder will ALWAYS use the latest for a given Flatcar channel and there's no way to override that.

Is that correct? (It shouldn't be but pretty sure we've missed this)

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-13 10:23:19

I'm pretty sure updating the filter to the following would fix things:

Flatcar{{env FLATCAR&#95;CHANNEL}}{{env FLATCAR&#95;VERSION}}**

Jeremi Piotrowski
2024-08-13 10:29:28

what you're sayingmakes sense to me

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-13 10:30:29

😞 Thanks for confirming. Guess I'll get a fix PR up shortly then

👍 Jeremi Piotrowski
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-13 10:37:47

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-13 15:04:23

Image-builder v0.1.32 is now available:
Thanks to all contributors! 🎉

GitHub
🎉 Drew Hudson-Viles, Abhay Krishna Arunachalam, Anurag, Sriraman Srinivasan
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-15 09:48:51

Image-builder v0.1.33 is now available:
Note: This is a small fix release to resolve the above issue with the latest Flatcar ISO release. There's not rush to upgrade to this if you don't use the Flatcar ISO. 🙂

GitHub
:partyk8s: Drew Hudson-Viles, chrischdi, Anurag, jsturtevant, Sriraman Srinivasan
Rende Luitjes
2024-08-19 10:54:20

@Rende Luitjes has joined the channel

Clark Zinzow
2024-08-21 07:42:02

@Clark Zinzow has joined the channel

Frans
2024-08-26 02:28:59

@Frans has joined the channel

Yongxiang Gao
2024-08-26 10:38:31

In ubuntu 22.04, we can find file /etc/default/grub, and there is line like this:

GRUB_CMDLINE_LINUX=" apparmor=1 security=apparmor"
How is it generated?
If I want to add something like "intel_iommu=on modprobe.blacklist=nouveau", how to do so within image-builder source code?

Sriraman Srinivasan
2024-08-26 11:51:20

If you have added the gpu role as part of your nodecustomrolespre and enabled the blocknouveau_loading variable (set to true ), nouveau driver will be blocklisted and kernel will not load the module. This disabling is done as part of gpu role.

GitHub
:ty: Yongxiang Gao
Yongxiang Gao
2024-08-26 16:52:21

I will try it out.
BTW, how about "inteliommu=on"?
Can this trick be use together with yours?
- name: Enable IOMMU
ansible.builtin.lineinfile:
path: /etc/default/grub
regexp: '^GRUB
CMDLINELINUXDEFAULT="((?:(?!inteliommu=on).)**?)"$'
line: 'GRUB
CMDLINELINUXDEFAULT="\1 intel_iommu=on"'
backup: true
backrefs: true
notify: update-grub

Sriraman Srinivasan
2024-08-27 07:02:32

  • Similar approach has been used for RedHat linux setting huge pages. You can do this in a custom role and invoke the role.

  • Also if not mistaken you can create a config file with the values in /etc/default/grub.d folder similar to the way we configure non-graphical consoles for debian systems.

  • Another way could be to update grub in boot config(cloud-init, etc) depending upon your OS. ---> This is something I have not tried, but could work(I may be wrong here)

GitHub
GitHub
Yongxiang Gao
2024-08-27 10:48:26

Thanks, yes, I'm able to add it as follows:
- name: Enable IOMMU
ansible.builtin.lineinfile:
path: /etc/default/grub
backup: true
backrefs: true
regexp: ^GRUBCMDLINELINUX="((?:(?!inteliommu=on).)**?)"$
line: GRUB
CMDLINELINUX="\1 inteliommu=on modprobe.blacklist=nouveau"
when: ansible_distribution == "Ubuntu"

Sriraman Srinivasan
2024-08-27 12:03:06

The above one will blacklist/block module from initially loading the module. However, if some other driver or something else tries to load it, it will still get loaded.

Yongxiang Gao
2024-10-21 06:30:21

What's the better way to add?

Yongxiang Gao
2024-10-21 09:43:59

Someone has luck to create EKSA vSphere cluster with ubuntu 22.04 at all?

Sriraman Srinivasan
2024-10-21 10:25:46

What's the better way to add?
Depends on whether you are likely to have a scenario where you have someother entity/driver trying to load nouveau kernel package. If yes and you wish to always make sure it does not load the package, you could add install nouveau /bin/false along with modprobe blocklisting. If thats not the case or does not matter if later package is loaded, what you have added should work.

:ty: Yongxiang Gao
Slackbot
2024-08-26 15:30:09

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma
2024-08-26 15:47:36

I'm not going to be able to attend this morning, but we probably have some things to discuss. If we don't have anything on the agenda and don't meet, I'll be here most of the day to discuss whatever on Slack.

Sriraman Srinivasan
2024-08-28 06:49:59

@mboersma /@Marcus Noble Can we close this issue since there has been no response?

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-28 07:07:08

Yeah I think so. Can always be reopened if needed

🙏 Sriraman Srinivasan
Scott Lowe
2024-08-28 23:45:25

@Scott Lowe has joined the channel

Scott Lowe
2024-08-28 23:51:49

Hey folks, I'm trying to use Image Builder to build an AWS AMI that will be used for an "offline" (no Internet access) K8s cluster. I need to load some additional container images into the AMI. To do that, I made the following changes to the files in my cloned copy of the Image Builder repo:

  1. In images/capi/ansible/roles/loadadditionalcomponents/defaults/main.yml, I set additionalregistryimages to "true".

  2. I provided a comma-delimited list of registry images in additionalregistryimages_list. The registry images are currently specified as "quay.io/path/image:tag".


Are there any other changes I need to make? Anything I'm missing from what you can tell?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 07:03:44

I vaguely recall some discussion about airgapped environments a few months ago. I can't remember the details though unfortunately but I you might be able to search this channels history.

👍:skin_tone_2: Scott Lowe
Scott Lowe
2024-08-29 15:07:08

Hmm, looks like the airgap discussion was about image-builder itself, not about building images for airgapped environments. 😕

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 15:10:36

Ah sorry. 😞

Scott Lowe
2024-08-29 15:11:42

No worries! I appreciate the response. Ignoring the airgap portion for a moment, do the changes I mention above sound correct for preloading additional container images into an AMI?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 15:12:15

I think so. But to be honest I'm not entirely sure as never used it myself.

Scott Lowe
2024-08-29 15:13:30

Fair enough. Thanks!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 15:14:16

Please report back how you get on! I'd be keen to hear if there was anything else.

Scott Lowe
2024-08-29 15:16:49

I will almost certainly write a blog post with details once I figure this out! ✍️

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 15:17:07

Perfect! 😄

Scott Lowe
2024-08-29 18:13:47

Here's the solution. All changes need to go into images/capi/packer/config/additionalcomponents.json. (No changes need to be made to the Ansible roles.) There are three changes to make:

  1. Set loadadditionalcomponents to "true".

  2. Set additionalregistryimages to "true".

  3. Specify a comma-delimited list of fully-qualified image names (i.e., "quay.io/path/image:tag") in additionalregistryimageslist.


The resulting AMI will have all the "base" container images as well as the additional container images you specified.

:parrotk8s: Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 07:00:57

Cross-posting this in case anyone else here has tried building the latest Flatcar on Azure yet. (Maybe @mboersma 🙏 )

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 11:11:46

github

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 11:56:10

☝️ @Drew Hudson-Viles if you're able to would you mind looking at this too?

👍 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 08:07:30

😩 To make things worse, ubuntu:latest now points to Ubuntu 24.04 which apparently doesn't have a qemu package available to install. Today is not going well for me 😅
Do y'all think we should pin the base ubuntu image to the previous (22.04)?

Drew Hudson-Viles
2024-08-29 08:57:40

I could be wrong here - but I think you have to install qemu-system since 23.xx

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 10:10:35

Yup, figured that out. 🙂

Then hit an issue where 24.04 now has the ubuntu user as 1000 and imagebuilder gets created as 1001 which broke all my permissions 😩

I'm not having a great morning so far.

Drew Hudson-Viles
2024-08-29 10:12:21

You need a weekend 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 10:18:18

I need a holiday I think 😅 This week has been A LOT

☝️ Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 10:19:14

But yeah, we might want to think about pinning the version as anyone building on top might experience similar issues. I suspect we should at least have it upgraded in a new release (maybe we pin to 24.04 and have that as a new release?)

👍 Sriraman Srinivasan
Drew Hudson-Viles
2024-08-29 10:23:26

yeah that sounds sensible to me

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 10:24:11

I'll try and get a PR up shortly. Currently trying to fix the above flatcar problem first as it's causing us problems 😞

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 10:38:36

Oh fun! I've also just learnt that Goss failing doesn't actually result in the make command failing 🤦‍♂️

:shake_fist: Drew Hudson-Viles, nikparasyr
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 11:17:58

github

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 13:02:33

🤦‍♂️ The PR also needs the /lgtm label. (No idea why when it has the approve 🤷 )

Sriraman Srinivasan
2024-08-30 05:36:17

Oh fun! I've also just learnt that Goss failing doesn't actually result in the make command failing
Realized that couple of days back when working on our systems using IB... goss validations failed but build succeeded.... wanted to raise a issue but forgot...

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 06:21:09

🤦‍♂️ i also forgot with everything else I was dealing with yesterday. I’ll create one today!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 08:00:29

🤦‍♂️ And I've just noticed thanks to writing up this issue that my CAPA builds have Goss failing

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 08:08:33

github

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 08:18:37

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 08:39:24

Will need to fix this first before handling the above:

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-29 17:07:00

Image-builder v0.1.34 is now available:
Thanks to all contributors! 🎉

Please note that this release include the above mentioned bumping of the base Ubuntu image from 22.04 -> 24.04.

GitHub
Brennen Murray
2024-08-29 22:54:55

@Brennen Murray has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 08:09:25

☝️ I don't really know much about Goss so not sure I can work on fixing it, at least not right now anyway, so if anyone is able to help I'd very much appreciate it! 💙

Edit: A few minutes searching and I think I've figured it out 😆

😄 Drew Hudson-Viles
Drew Hudson-Viles
2024-08-30 09:02:02

You're too ninja fast for me. I log in and you've both found a problem and may have a fix 😄

I don't have an Amazon env to build these in unfortunately either. But I can take a look from the OpenStack perspective if needed still.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 09:03:41

It looks like its Flatcar specific. It's just I had logs saved for my previous AMI builds 🙂 (Edit - the failing goss test is flatcar specific)

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 09:03:53

Anyway, I have a fix PR incoming 😉

👍 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 09:05:48

Fix for the Flatcar failure:
Ensure Goss failure fail the image build:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 09:06:21

(I'm having a MUCH more productive day so far than I did yesterday 😆 )

😆 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 09:10:08

I also feel like I might finally be starting to get a grip on how most of image-builder fits together 🤣

Drew Hudson-Viles
2024-08-30 09:12:37

Yeah I feel like I'm finally there now. The attempt to switch to HCL (which I've put on hold until we know where we stand with packer) made me learn a significant amount about the structure and more importantly the beast of a Makefile.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-08-30 09:14:11

Oh the Makefile is still a "here be dragons" thing for me. I can tweak it a little but I wouldn't feel comfortable making any large changes to it 😅

😄 Drew Hudson-Viles, Sriraman Srinivasan
Gareth
2024-09-01 05:57:07

@Gareth has joined the channel

Karthik-K-N
2024-09-03 07:38:41

@Karthik-K-N has joined the channel

Karthik-K-N
2024-09-03 07:53:51

Hi Team, Trying to understand about previous work or future plan for supporting fedora based image building for CAPI. Could some one please help.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-03 07:55:35

What are you wanting to know exactly?

There is no dedicated support to any OS or provider and we rely on community contributions. Are you looking to help out with Fedora support?

Karthik-K-N
2024-09-03 07:56:41

Yes, I am looking forward to use fedora for creating a CAPI cluster

Karthik-K-N
2024-09-03 07:57:37

I see that currently image-builder does not have support for building fedora images, So looking forward to understand about plan for adding fedora support or how can I add the support

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-03 08:01:41

First place to start would likely be to open an issue on the repo stating what versions of Fedora you're looking for and against what providers.

I know we have CentOS support for AWS at least so that might be a good place to look at what might be needed for adding Fedora.

Karthik-K-N
2024-09-03 08:02:56

Sure, Thanks for the quick help. I will create an issue and possible try to add fedora support as well.

👍 Marcus Noble
Karthik-K-N
2024-09-03 10:00:19

Created issue for reference

GitHub
:thank_you_icon: Marcus Noble
Karthik-K-N
2024-09-04 05:22:45

@Marcus Noble By any chance do you have any doc or reference or PR on what are the things needed to be done to add a new OS support. Just looking for some reference

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-04 05:31:59

We dont have any docs unfortunately (we prob should 😔 ) but I could likely find an example PR for you. Let me see what I can find…

Karthik-K-N
2024-09-04 05:32:20

sure that will be really helpful.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-04 05:38:51

Here’s a pretty large PR adding a totally new OS distro: https://github.com/kubernetes-sigs/image-builder/pull/1192

And here’s a couple small ones just adding newer versions of existing distros:
https://github.com/kubernetes-sigs/image-builder/pull/1500
https://github.com/kubernetes-sigs/image-builder/pull/1476

Karthik-K-N
2024-09-04 05:39:56

Nice, Thank you I will use this as a reference.

Kepler SysAdmin
2024-09-04 17:40:30

When Define variables for OpenStack build,
What version are available for variable kubernetes_version?
ref -

Drew Hudson-Viles
2024-09-04 17:46:44

It just depends which version of Kubernetes you want. You can find a list on the Kubernetes releases page.

Kepler SysAdmin
2024-09-04 17:47:24

here: ?
What happens if i want an version that is not longer in the list?

Drew Hudson-Viles
2024-09-04 17:48:44

It likely won't be supported as image builder is configured to use the new repos which iirc started at 1.27 or 1.28. Also, they likely aren't maintained any more so it wouldn't get the latest updates or security patches.

Kepler SysAdmin
2024-09-04 17:49:25

What is the list?
This: ?

Kepler SysAdmin
2024-09-04 17:57:37

basically what are the supported version when using the image builder?

Drew Hudson-Viles
2024-09-04 18:20:51

Yeah that's the list. The only ones we can really support are the same as are listed in that link as anything further back is not supported by the Kubernetes community

👍 Kepler SysAdmin
Kepler SysAdmin
2024-09-04 21:34:36

Thank you!

Daz Mac
2024-09-09 05:02:48

@Daz Mac has joined the channel

Slackbot
2024-09-09 15:30:20

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles
2024-09-09 16:02:14

Anything to discuss this week? I know we have one ongoing item but do we need to catch up today on this?

Drew Hudson-Viles
2024-09-09 16:37:55

Skipping this week due to no attendance

vikram
2024-09-11 05:53:32

@vikram has joined the channel

Anil Hariharan
2024-09-11 06:03:41

@Anil Hariharan has joined the channel

Anil Hariharan
2024-09-11 06:13:44

Hi, we are currently trying to generate ubuntu 22.04 image with kubernetes v1.28.0 but keep running into problems. Can somebody please help?

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-11 07:05:09

I’m no expert on qemu but this line looks relevant:

2024/09/10 09:37:30 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/09/10 09:37:30 Qemu stderr: Unable to init server: Could not connect: Connection refused
Maybe some sort of permission or connection issue? I haven’t seen this before myself.

Anil Hariharan
2024-09-11 07:32:39

im able to build ubuntu 20.04 with k8s v1.28, but run into error when its ubuntu 22.04 with k8s v1.28

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-11 07:48:38

Oh that’s weird. I’m not aware of anything that might be different unfortunately 😔 Hopefully someone else has some suggestions. I’m currently travelling so not able to test myself currently but @Drew Hudson-Viles might be able to if he’s about this week? 🤞

Drew Hudson-Viles
2024-09-11 08:39:16

sure thing, let me take a look

💙 Marcus Noble
Drew Hudson-Viles
2024-09-11 08:59:56

I've managed to build this myself using the same command. the only difference is I didn't run it with sudo. I get a different error when running with sudo as I don't have the packages available in the PATH. It is definitely worth checking if this is the source of the problem though. But I can confirm the process definitely works.

I've dropped a commend in the issue anyway.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-11 09:00:29

Might be worth trying with the container image too is possible

Drew Hudson-Viles
2024-09-11 09:00:37

Indeed

Anil Hariharan
2024-09-11 13:53:34

even without sudo it gets stuck at qemu: Waiting for SSH to become available...

Drew Hudson-Viles
2024-09-11 13:54:54

I'd recommend launching the vnc and checking what's happening on the VM it's launched in this case. I've ran through this on my side and I can confirm it's 100% working.

The other option would be to try using the Docker image that's supplied to run a build.

Anil Hariharan
2024-09-13 05:36:10

we were able to generate the image now, Thanks!

🙌 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-15 10:03:03

What was the solution in the end?

Anil Hariharan
2024-09-16 05:43:24

something wrong with the dev environment we were working on, tried it from scratch on a new machine and it worked!

mboersma
2024-09-19 18:21:38

Image-builder v0.1.35 is now available: https://github.com/kubernetes-sigs/image-builder/releases/tag/v0.1.35
Thanks to all contributors!

GitHub
:emoji_k8s_loft: Drew Hudson-Viles
Abhay Krishna Arunachalam
2024-09-20 04:28:17

Hello folks, I have opened a PR to fix a Goss bug that caused our RHEL 8 and 9 qemu/raw builds to fail due to validation errors. I have updated the PR description with the necessary information about the bug. I would greatly appreciate some reviews on it, thank you!

:thank_you_icon: Marcus Noble
👍 mboersma
KJ
2024-09-20 17:53:39

@KJ has joined the channel

Omar
2024-09-22 06:49:20

@Omar has joined the channel

Slackbot
2024-09-23 15:30:09

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-23 15:39:49

Agenda empty - shall we still meet or skip? Anyone got anything to discuss?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-23 15:51:22

I think I'm going to skip. I could do with a rest as not been able to focus at all today 😩 If anyone decides to sync then ping me and I'll try to join.

❤️ mboersma
mboersma
2024-09-23 15:58:05

Sounds good to me, sorry to chime in late. I don't have anything for the agenda but if anyone does, I'll check in a bit and start the meeting if so.

👍 Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-26 10:52:41

Is anyone building CAPA images that can pull from a private ECR registry?
If so - how are you including the ecr-credential-provider in your AMIs now that it's not included in-tree? (Related issue - )

Danny Seymour
2024-09-27 00:32:11

I'm actually getting all of this set up right now. I basically wrote a custom python script to generate packer vars including the correct ECR provider version and then wrote a role to install the provider.

Danny Seymour
2024-09-27 00:34:38

I elected to put the logic inside a Python script because I'm not good enough with Ansible to navigate the Github releases API with it and choose the appropriate version.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-27 06:15:38

I’m attempting to get it added into image-builder behind a Boolean toggle. Getting a bit tangled up with the ansible vars but hoping to have something today.

Danny Seymour
2024-09-27 07:22:50

Ooooo...I'd love to see that.

Danny Seymour
2024-09-27 07:23:11

If you want a tester, sign me up.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-27 07:23:32

Well... hopefully I get it working today 🤞 We're currently blocked on an upgrade without it so I need to figure something out.

Danny Seymour
2024-09-27 07:25:50

Heh...I know the feeling. We've been blocked on getting upgraded past 1.28.3 due to the whole VMWare acquisition led to the discontinuance of CAPI images. I'm trying to get a whole build pipeline automated so we can quit worrying about winding up outside the supported version window.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-27 07:26:49

Yeah, we build all our images too. But mainly because we wanted consistency across different providers that we support. It's constantly a work in progress it seems 🙈

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-27 10:39:13

Ok, at first glance, my changes are looking good:

/ # ls -la /host/opt/bin/ecr-credential-provider
-rwxr-xr-x 1 root root 477197 Sep 27 09:08 /host/opt/bin/ecr-credential-provider
/ # ls -ls /host/var/usr/ecr-credential-provider/ecr-credential-provider-config
4 -rw-r--r-- 1 root root 337 Sep 27 09:08 /host/var/usr/ecr-credential-provider/ecr-credential-provider-config
/ # cat /host/var/usr/ecr-credential-provider/ecr-credential-provider-config
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
- name: ecr-credential-provider
matchImages: ['.dkr.ecr..amazonaws.com', '.dkr.ecr..amazonaws.com.cn']
defaultCacheDuration: "12h"
apiVersion: credentialprovider.kubelet.k8s.io/v1
env:
- name: AWS_PROFILE
value: "default"
/ # ps ax | grep /opt/bin/kubelet
3225 root 0:02 /opt/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cloud-provider=external --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --healthz-bind-address=0.0.0.0 --v=2 --image-credential-provider-config=/var/usr/ecr-credential-provider/ecr-credential-provider-config --image-credential-provider-bin-dir=/opt/bin

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-27 10:39:33

Just need to get another team to test it out and make sure it actually works with ECR then I'll get a PR up.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-27 16:49:01

Confirmed working 😄 PR is up

GitHub
Danny Seymour
2024-09-27 20:25:11

I did ask a question on the PR after it got merged. It's a bit of a nit, but I figure it's probably better to ask.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-27 20:30:29

I wasn’t actually aware it was available at that URL. I struggled to find docs about it. If that URL works the same then yeah it makes sense to use that instead. Do you want to open a PR for it?

Danny Seymour
2024-09-27 20:35:45

Sure

Danny Seymour
2024-09-27 20:36:13

It took me a while to find it too when I went searching.

Danny Seymour
2024-09-27 21:05:32

This is the first reference I could find for it related to sigs:

GitHub
Danny Seymour
2024-09-27 21:08:06

GitHub
💙 Marcus Noble
:thank_you_icon: Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-30 07:35:19

I was considering pushing out a new release with this change but I think I'm going to hold off until there's more changes that need releasing. Just, FYI 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2024-09-27 19:44:10

Image-builder v0.1.36 is now available:

This include the above change I was discussing to optionally include ecr-credential-providers.

GitHub
:thank_you: mboersma
🎉 mboersma
Arnaud Pons
2024-10-01 07:30:21

@Arnaud Pons has joined the channel

Ian King’ori
2024-10-01 12:09:19

@Ian King’ori has joined the channel

TinaMor
2024-10-01 16:52:57

@TinaMor has joined the channel

Yongxiang Gao
2024-10-04 07:51:36

I try to build vSphere image with such command:

IB_OVFTOOL_ARGS="--allowExtraConfig" make build-node-ova-vsphere-ubuntu-2204-efi

During the build process, we can see following message:
==> vsphere-iso.vsphere: Power on VM...
==> vsphere-iso.vsphere: Waiting 10s for boot...
==> vsphere-iso.vsphere: HTTP server is working at
==> vsphere-iso.vsphere: Typing boot command...
==> vsphere-iso.vsphere: Waiting for IP...
==> vsphere-iso.vsphere: IP address: 10.20.34.127
==> vsphere-iso.vsphere: Using SSH communicator to connect: 10.20.34.127
==> vsphere-iso.vsphere: Waiting for SSH to become available...

The VM was 10.20.34.127 initially, however, very soon, it reboot and the IP change to something like 10.20.34.128, then "Waiting for SSH to become available..." will stuck there forever (because it's waiting for the old IP).

2004 doesn't have this issue.

How to avoid this 2204 specific issue?

Yongxiang Gao
2024-10-04 08:12:42

Same issue exists for the command:

image-builder build --os ubuntu --os-version 22.04 --hypervisor vsphere --release-channel 1-29 --vsphere-config vsphere-34.json --firmware efi

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-04 08:18:38

I don't have an answer myself but does this help at all?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-04 08:18:57

Looks like you might be able to force it to use a specific IP address at boot. Not sure how we solve this permanently though.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-04 08:19:43

Or possibly this:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-04 08:20:02

It might be we're not waiting long enough for it to detect the new IP address from vsphere

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-04 08:21:57

Try setting boot_wait to 300 in your packer vars and see if that helps at all. If it does we could try tweaking it until we find a low enough value that works.

Yongxiang Gao
2024-10-04 08:56:25

Thanks, I try following 3 ways in file packer/ova/packer-node.json:

  1. change boot_wait to 300 at line 42

  2. add at the front of line 38

  3. add at the end of line 40


None of them helps.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-04 08:57:25

Can you confirm that in the vcenter console it is picking up the new IP address?

Yongxiang Gao
2024-10-04 08:59:44

Yes, in vcenter, the new IP shows up very soon.

Yongxiang Gao
2024-10-04 09:00:04

no 5 minute wait at all.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-04 09:00:51

Hmmmm... I thought that was how Packer discovered the IP. Maybe someone who knows more about vsphere can suggest something.

Danny Seymour
2024-10-04 23:03:39



It looks like this may be caused by the behavior of the DHCP server in use.

GitHub
Danny Seymour
2024-10-04 23:12:36

Ah, that's one of the things that Marcus is getting at with the boot_wait.

Yongxiang Gao
2024-10-10 18:50:40

The trick doesn't help (the IP still changed) with the image-builder build from source if put in late-commands.
Instead, if put in early-commands, it helps:

images/capi# git diff packer/ova/linux/ubuntu/http/22.04.efi/user-data
diff --git a/images/capi/packer/ova/linux/ubuntu/http/22.04.efi/user-data b/images/capi/packer/ova/linux/ubuntu/http/22.04.efi/user-data
index 095d9cef3..ee9bb9705 100644
--- a/images/capi/packer/ova/linux/ubuntu/http/22.04.efi/user-data
+++ b/images/capi/packer/ova/linux/ubuntu/http/22.04.efi/user-data
@@ -20,6 +20,9 @@ autoinstall:
# Disable ssh server during installation, otherwise packer tries to connect and exceed max attempts
early-commands:
- systemctl stop ssh
+ # Prevent DHCP release message from being sent on reboot
+ - iptables -A OUTPUT -p udp --dport 67 -j DROP

images/capi# IB_OVFTOOL_ARGS="--allowExtraConfig" PACKER_FLAGS="--var 'kubernetes_rpm_version=1.28.9' --var 'kubernetes_semver=v1.28.9' --var 'kubernetes_series=v1.28' --var 'kubernetes_deb_version=1.28.9-2.1'" make build-node-ova-vsphere-ubuntu-2204-efi
...
==> vsphere-iso.vsphere: IP address: 10.20.34.145
==> vsphere-iso.vsphere: Using SSH communicator to connect: 10.20.34.145
==> vsphere-iso.vsphere: Waiting for SSH to become available... ** the IP stays at 10.20.34.145
...

However, is there side-effects?

Yongxiang Gao
2024-10-11 16:56:32

If there is side effect, what's the right way to fix the side effect?

Sriraman Srinivasan
2024-10-17 06:31:04

Try to increment ipsettletimeout and ipwaittimeout .

Yongxiang Gao
2024-10-21 06:27:58

@Sriraman Srinivasan Where to increase? Please share a diff.

Sriraman Srinivasan
2024-10-21 10:33:06

  • You can create a custom json file

  • Add ipsettletimeout to the json and set it to say 15m or adjust according to time for boot.

  • Add path of the json file to PACKERVARFILES environment and run OVA build

Sriraman Srinivasan
2024-10-21 13:04:46

ipsettletimout will make packer wait for specified time duration before it asks for the VM IP address.

:ty: Yongxiang Gao
Will Foster
2024-10-04 19:28:04

@Will Foster has joined the channel

leland knight
2024-10-06 00:18:08

@leland knight has joined the channel

leland knight
2024-10-06 02:01:58

I've got a question around image-builder-proxmox, where can I ask about the proxmox image builder?

==> proxmox-iso.ubuntu-2204:  => downloaded_iso_path/c968bbbeb22702b3f10a07276c8ca06720e80c4c.iso
==> proxmox-iso.ubuntu-2204: 501 for data too large
Build 'proxmox-iso.ubuntu-2204' errored after 2 minutes 31 seconds: 501 for data too large
==> Wait completed after 2 minutes 31 seconds
==> Some builds didn't complete successfully and had errors:
--> proxmox-iso.ubuntu-2204: 501 for data too large
I saw an older posting with this error and it sounded like it had something to do with the Mime-type used when uploading.

Additional detailed added to a ticket:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-07 07:07:06

I've only used the Proxmox provider a little bit for some testing so might not be able to help but I do remember it being quite a pain to get set up.

Can you share what make target you're using and what vars you're providing (with anything sensitive redacted)

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-07 07:08:21

Also, do you have any load balancer or proxy or anything in front of your proxmox api that might be imposing the limit? (e.g. nginx)

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-07 07:10:22

I can't remember exactly but I think I ended up manually downloading the base iso into Proxmox prior to running image-builder

leland knight
2024-10-07 17:11:58

I also ended up manually uploading an iso to proxmox, and I thought it helped at first, but then didn't seem to. Figured I was just starting to hack at it rather than fix the actual issue.

I wonder if you might click on the link I provided, as I put a bunch of additional detail in the ticket 288.

leland knight
2024-10-10 02:40:57

answer:

  1. 501 is because I needed to add /api2/json

  2. "using closed network connection" was due to 403


in my case this was difficult to diagnose because i was granting the correct access to an api token on the storage needed; but i hadn't granted the user the needed access which the api token came from; i added permissions to the user then the storage access worked

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-10 03:32:26

Are you able to suggest any improvements to our documentation?

leland knight
2024-10-10 03:35:14

I'm thinking about it ... figured I should probably first get things working then maybe go back and add something. So far I'm thinking:

  • it'd be nice if those two errors could be more descriptive so the person running into them could be faster to resolve them, i put a couple suggestions in the issue i closed

  • i went back to the proxmox instructions and it was clear to add permissions to the user (not the api token), simply i install proxmox a few days ago for the first time ... so that wasn't enough for me to avoid the issue i ran into; it would be nice to maybe have an additional step that lists out the permissions of the users and greps for the required permission ... something like that because as the person doing it i would have then realized i didn't make the user correctly

leland knight
2024-10-10 03:36:04

that's what i did when i fixed them; i listed out the user permissions; added the role; then listed them out again to see that they had changed

leland knight
2024-10-10 03:36:28

(cause at that point i wasn't sure if i had added the permissions before or not, just wasn't sure)

leland knight
2024-10-10 03:36:50

if i have time i'll put together a youtube video maybe, will let you know if i do

leland knight
2024-10-10 03:43:10
Perhaps there is an opportunity to make the process easier for folks in the future in the documentation, and perhaps in the error messages. Instead of 501 maybe "501, did you remember /api2/json?" Instead of "use of closed network connection" maybe "use of closed network connection (403)" or "use of closed network connection, see /var/log/pveproxy/access.log for more detail".
leland knight
2024-10-10 03:51:39

if someone forgets /api2/json, maybe it could just add it automatically ... or the URL environment variable could be checked early on to see if it has '/api2/json' and fail quick if it doesn't

leland knight
2024-10-10 04:43:11

also, how to specify which kubernetes version to build, the doc page () lists 4 env variables ... but i wonder if we actually just need one ... like KUBERNETES_SEMVER ... or do we need to specify all 4; when i run the build without specifying any KUBERNETES variables i don't really know what's going to happen? it will probably build the latest kubernetes, i assume? but don't know

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-10 06:55:27

That error message comes directly from Packer so not something we can really control in image-builder unfortunately. We could maybe do a check for the api2 path though. The example on the docs page does include the path though but maybe we need to make it clearer somehow.

If you have any suggestions on how to make those docs better we'd very much appreciate a PR! 😄 None of the maintainers of image-builder use Proxmox so we rely heavily on user contributions here.

I answered the kube version question in your other thread 🙂

Justin Holmes
2024-10-06 21:25:06

@Justin Holmes has joined the channel

Slackbot
2024-10-07 15:30:12

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-07 15:37:07

Agenda currently empty. If no one has anything they'd like to discuss then lets skip 🙂

mboersma
2024-10-07 15:49:39

I’ll have to miss it regardless today, sorry! Taking our animals to the vet for a checkup.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-07 15:50:41

ok lets skip 🙂

👍 mboersma
Drew Hudson-Viles
2024-10-07 16:40:51

Sorry, I've only just seen the time. The whole family is ill and the lack of sleep has made today just wizz by 😄

❤️ mboersma
mboersma
2024-10-07 16:48:01

Hang in there Drew!

🙏 Drew Hudson-Viles
richcase
2024-10-09 08:55:13

Should i be worried about these warnings in the image-builder logs? Skipped '/run/netplan' path due to this access issue

Drew Hudson-Viles
2024-10-10 19:31:02

I've not come across this one myself but it may just be the path isn't available if it's skipping over. As long as the image is coming online on boot, I wouldn't be too concerned.

richcase
2024-10-11 15:56:34

Thanks @Drew Hudson-Viles. The image does come online.

👍 Drew Hudson-Viles
leland knight
2024-10-10 02:42:38

Where does "proxmox-iso.ubuntu-2404" come from? I mean, where is it built so I can see what's in there maybe tweak it a little. Looks like there is a bug and I was interested to look into it a bit. (Or is that the new vm being built? It said ISO so thought it might have been an iso)

==> proxmox-iso.ubuntu-2404: Error creating VM: format can only be one of the following values: cow,cloop,qcow,qcow2,qed,vmdk,raw
Build 'proxmox-iso.ubuntu-2404' errored after 3 minutes 31 seconds: Error creating VM: format can only be one of the following values: cow,cloop,qcow,qcow2,qed,vmdk,raw

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-10 06:46:43

It should pull it from here I think:

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-10 06:47:06

If I understand your question correctly.

leland knight
2024-10-10 22:22:26

I'm was hoping to troubleshoot the error maybe ... but I'm too in the dark on this one. Thank you. I'll just hang out and hope the issue 1579 gets some attention.

leland knight
2024-10-10 04:33:55

How can I automatically set the PACKER kubernetes versions? I feel like I'm reinventing the wheel here ... and not sure how to get the DEB version:

# parse and set kubernetes env vars
echo "PACKER_FLAGS: $PACKER_FLAGS"
export VERSION=v1.31.1

#KUBERNETES_RPM_VERSION=1.29.6
#KUBERNETES_SEMVER=v1.29.6
#KUBERNETES_SERIES=v1.29
#KUBERNETES_DEB_VERSION=1.29.6-1.1

KUBERNETES_RPM_VERSION=$(echo $VERSION | cut -d 'v' -f 2)
KUBERNETES_SEMVER=$VERSION
KUBERNETES_SERIES=$(echo $VERSION | cut -d '.' -f 1).$(echo $VERSION | cut -d '.' -f 2)
KUBERNETES_DEB_VERSION=$(echo $VERSION | cut -d 'v' -f 2)-1.1

export PACKER_FLAGS="--var kubernetes_rpm_version=$KUBERNETES_RPM_VERSION --var kubernetes_semver=$KUBERNETES_SEMVER --var kubernetes_series=$KUBERNETES_SERIES --var kubernetes_deb_version=$KUBERNETES_DEB_VERSION"

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-10 06:50:16

You can define your own vars JSON file and pass that in when calling image-builder.
E.g. I have a vars.json that looks something like this (with the env vars populated via shell fist):

{
"ssh_clear_authorized_keys": "true",
"kubernetes_deb_version": "${KUBERNETES_VERSION}-00",
"kubernetes_rpm_version": "${KUBERNETES_VERSION}-0",
"kubernetes_semver": "v${KUBERNETES_VERSION}",
"kubernetes_series": "v${VERSION_MAJOR}.${VERSION_MINOR}",
"enable_containerd_audit": "true",
"ecr_credential_provider": "true"
}
And then set the environment variable PACKERVARFILES to the location of that vars.json when calling image-builders Make targets

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-10 06:51:31

Unfortunately its not quite as easy as just saying "give me Kubernetes v1.31" as the different platforms/OSs have different ways of pulling them in and we don't have the capacity in the project to keep an up-to-date mapping of those versions.

Peter Lindblom
2024-10-11 08:33:31

@Peter Lindblom has joined the channel

mcbenjemaa
2024-10-11 12:03:10

Hey there,

I have new PR:



/cc @leland knight
can you test my PR on your setup, to double check.
It is actually working for me.

leland knight
2024-10-11 15:22:13

It's getting further along now. At the moment I'm seeing this message:

Error getting SSH address: 500 QEMU guest agent is not running

leland knight
2024-10-11 15:52:55

With proxmox build now seeing:

Error getting SSH address: 500 QEMU guest agent is not running
(dhcp is enabled and works)

The console shows:

mcbenjemaa
2024-10-11 16:25:19

The packer VM couldn't access the packer HTTP server.

You need to run this in Proxmox VM (same network) for example, so you're sure that communication works

leland knight
2024-10-11 16:29:18

are we sure the qemu guest agent is getting installed by default?

leland knight
2024-10-11 16:32:17

i'll verify i'm able to reach the port from a remote system...

leland knight
2024-10-11 16:38:11

all tests showing network connectivity is good and no firewall issues; able to curl the port image-builder opens up ; gives me a 403, but that's ok, communication is good

leland knight
2024-10-11 16:39:00

i stopped a firewall to be completely sure, so i'll let it run for a bit with the 500 ... but i suspect the communication should happen pretty quick

leland knight
2024-10-11 17:00:41

still says waiting on 500, but the console isn't showing the startup screen anymore, i see automation in there so it seems to be working ... acting like its stuck on 'installing kernel' ... fingers crossed it works

leland knight
2024-10-11 17:14:16

things were looking good, finished this time but with an error:

2024/10/11 10:09:01 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/11 10:09:01 [INFO] 0 bytes written for 'stdin'
proxmox-iso.ubuntu-2204:
proxmox-iso.ubuntu-2204: TASK [Gathering Facts] *
proxmox-iso.ubuntu-2204: fatal: [default]: FAILED! => {"msg": "failed to transfer file to /home/travis/.ansible/tmp/ansible-local-350766c889qert/tmph0vrydqs /tmp/.ansible/ansible-tmp-1728662940.5550468-350781-30051397384999/AnsiballZ_setup.py:\n\n"}
proxmox-iso.ubuntu-2204:
proxmox-iso.ubuntu-2204: PLAY RECAP

proxmox-iso.ubuntu-2204: default : ok=1 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0
proxmox-iso.ubuntu-2204:
2024/10/11 10:09:01 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/10/11 10:09:01 shutting down the SSH proxy
2024/10/11 10:09:01 [INFO] (telemetry) ending ansible
==> proxmox-iso.ubuntu-2204: Provisioning step had errors: Running the cleanup provisioner, if present...
==> proxmox-iso.ubuntu-2204: Stopping VM
==> proxmox-iso.ubuntu-2204: Deleting VM
2024/10/11 10:09:05 [INFO] (telemetry) ending ubuntu-2204
==> Wait completed after 23 minutes 34 seconds
2024/10/11 10:09:05 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2024/10/11 10:09:05 machine readable: proxmox-iso.ubuntu-2204,error []string{"Error executing Ansible: Non-zero exit status: exit status 2"}
==> Builds finished but no artifacts were created.
2024/10/11 10:09:05 [INFO] (telemetry) Finalizing.
Build 'proxmox-iso.ubuntu-2204' errored after 23 minutes 34 seconds: Error executing Ansible: Non-zero exit status: exit status 2

==> Wait completed after 23 minutes 34 seconds

==> Some builds didn't complete successfully and had errors:
--> proxmox-iso.ubuntu-2204: Error executing Ansible: Non-zero exit status: exit status 2

==> Builds finished but no artifacts were created.
2024/10/11 10:09:05 waiting for all plugin processes to complete...
2024/10/11 10:09:05 /home/travis/.config/packer/plugins/github.com/hashicorp/proxmox/packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64: plugin process exited
2024/10/11 10:09:05 /home/travis/.config/packer/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/10/11 10:09:05 /home/travis/.config/packer/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/10/11 10:09:05 /usr/bin/packer: plugin process exited
2024/10/11 10:09:05 /usr/bin/packer: plugin process exited
2024/10/11 10:09:05 /usr/bin/packer: plugin process exited
2024/10/11 10:09:05 /home/travis/.config/packer/plugins/github.com/YaleUniversity/goss/packer-plugin-goss_v3.2.12_x5.0_linux_amd64: plugin process exited
make:
* [Makefile:593: build-proxmox-ubuntu-2204] Error 1

mcbenjemaa
2024-10-11 17:23:18

Its just some ansible issue.
Make sure everything fine.

leland knight
2024-10-11 17:36:15

Is the end result a randomly named iso in the storage?

mcbenjemaa
2024-10-11 17:36:42

Proxmox template

leland knight
2024-10-11 17:37:11

it should be named proxmox template?, or ... you are saying the randomly named iso is the proxmox template?

first time i'm using this, not even sure what the result is supposed to be, not sure if this is a successful run or not since i got the ansible error

leland knight
2024-10-11 17:38:50

I would have expected something like, "Success! Generated xxx.iso".

mcbenjemaa
2024-10-11 17:39:51

What are you talking about?
What ISO
It should be Virtual machine template in Proxmox.

leland knight
2024-10-11 17:45:38

I don't see a template, I think after the ansible error it just shutdown the vm and deleted it.

==> proxmox-iso.ubuntu-2204: Provisioning step had errors: Running the cleanup provisioner, if present...
==> proxmox-iso.ubuntu-2204: Stopping VM
==> proxmox-iso.ubuntu-2204: Deleting VM

mcbenjemaa
2024-10-11 17:46:36

Thats because the build was failed

leland knight
2024-10-11 18:02:13

I don't know how to troubleshoot this, so just put in another issue ...

mcbenjemaa
2024-10-11 18:10:57

Please create only one issue.
And comment there.

mcbenjemaa
2024-10-11 18:12:41

Again it seems some ssh or scp is failing in your setup.
Try to check your firewall whatsoever
or anything you need to add to ansible ssh or scp

leland knight
2024-10-12 01:14:48

The firewall is still disabled, so that shouldn't be the issue ... I'm not sure what else would be needed

leland knight
2024-10-12 01:16:25

What is it trying to transfer and from where to where?

Mike Tritabaugh
2025-06-11 14:58:57

Adding my two cents in case anyone else runs into this error and finds this thread…

I got this error after successfully building multiple images. Nothing obvious had changed, I even tried rebuilding images that previously worked.

I resolved it by updating the ubuntu-2404.json file. I changed the boot command to hard code my laptop’s IP address (where I’m running the make command).

  "boot_command_prefix": "clinux /casper/vmlinuz --- autoinstall ds='nocloud-net;s=:{{ .HTTPPort }}/24.04/'initrd /casper/initrdboot",

My uneducated hypothesis:
After setting PACKER_LOG=1, I noticed that the http server was starting on 0.0.0.0 . This means all interfaces, but I suspect this IP is being passed to the build VM as a callback, which obviously wouldn’t work. After making the change above, the builds completed successfully. I’m assuming something changed in my environment (it had been a couple weeks since I ran a build) that caused the http server to change the bind IP used.

Or, I could be completely wrong. 🤷‍♂️

leland knight
2024-10-11 15:54:14

[ MESSAGE TEXT EMPTY ]

leland knight
2024-10-11 16:13:41

Issue filed:

richcase
2024-10-11 18:01:50

If anyone is free i could do with a review on this change:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-11 18:29:39

Thanks! I thought it was a duplicate of https://github.com/kubernetes-sigs/image-builder/pull/1586 at first but looks like it’s different Ubuntu versions. Do you know if any others need changing?

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-11 18:30:06

Is this just an issue on AWS?

richcase
2024-10-11 18:35:42

Its just an AWS issue as far as i know as CAPA does some hacky things with cloud-init

richcase
2024-10-11 18:36:07

We're working on a longer term solution so that we don't need to pin to an old version.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-11 18:48:27

👍 great! Are you needing a new release of image-builder putting out then?

richcase
2024-10-11 18:49:12

I'm just using main at the moment.

richcase
2024-10-11 18:49:34

But thinking about it we should pin our build pipeline to a version.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-11 18:50:06

Ok. If you’re ok for now I can hold off until Monday to get a new release out.

richcase
2024-10-11 18:50:28

Sounds good

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-11 18:50:51

Unless I find a bit of time tomorrow. I might kick off the process as there’s been a few fixes the past week.

👍 richcase, leland knight
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-12 13:10:21

😞 Looks like I'm actually unable to push a new release out right now -

Marcus Noble (https://kubernetes.slack.com/team/U9X94MGUB)
:ack: richcase
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-12 18:49:30

Image-builder v0.1.37 is now available:

Some notable changes:

  • This release fixes for building Ubuntu images on CAPA by pinning the version of cloud-init deb

  • The default Kubernetes version is updated to v1.30.5

  • The ability to set the isofile or isourl when building Proxmox images proxmox

  • Adds Windows Server 2025 to Azure azure

  • Some refactoring and cleanup of the code 🧹


Thanks to all contributors! 🎉

GitHub
🎉 Drew Hudson-Viles, leland knight, richcase
leland knight
2024-10-12 22:11:04

I'd like to run image builder via a gitlab runner. I know what IP it will use, but being that its running in kubernetes its exposed via a service, or even better an ingress and I have the FQDN. Where can I specify the IP to use or specify FQDN? or otherwise tell image-builder to use the loadbalancerip when it calls back to the image-builder http port?

Submitted as a feature request

Continuing to search through docs to see if the feature already exists and I just haven't found the right env vars yet.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-14 16:25:47

Image-builder v0.1.38 is now available:

⚠️ Important
This release contains fixes for two CVEs - CVE-2024-9486 and CVE-2024-9594 (see kubernetes/kubernetes#128006 & kubernetes/kubernetes#128007 for more details).
It is highly recommended to update your version of image-builder and re-build all your VM images.

GitHub
:thank_you: mboersma, Drew Hudson-Viles, Abhay Krishna Arunachalam, mcbenjemaa
🎉 mboersma, Drew Hudson-Viles
:partyk8s: mboersma, Drew Hudson-Viles, leland knight
:kubernetes_intensifies: Drew Hudson-Viles
Abhay Krishna Arunachalam
2024-10-14 17:38:11

Hello Image-builder maintainers, we just migrated to v0.1.38 to fix the CVEs and our Nutanix builds started failing with the error

==> nutanix: Waiting for SSH to become available...
==> nutanix: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
==> nutanix: Force deleting virtual machine...
nutanix: Virtual machine successfully deleted
I am actively looking into it, but would appreciate if any of you folks might know what's causing the builds to act up. Maybe some hard-coded credentials needs to be removed?

Abhay Krishna Arunachalam
2024-10-14 17:39:51

Seeing the same for Vsphere ISO

==> vsphere-iso.vsphere: Waiting for SSH to become available...
==> vsphere-iso.vsphere: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

Drew Hudson-Viles
2024-10-14 17:49:48

I'm wondering if it's this change and that uuid is being overridden.

https://github.com/kubernetes-sigs/image-builder/pull/1596/files#diff-19301ae03119dcb0a5ed81f1e5839a3a26725e486cb89a3cc3e70c1b1df0b159R160

Can you confirm your vars for me? Do you have a password set in there that configures this value already?

GitHub
Abhay Krishna Arunachalam
2024-10-14 17:51:50

no we don't set the password manually, have always resorted to the builder/builder default, and now we've taken in this commit without any patches to it

Abhay Krishna Arunachalam
2024-10-14 17:59:27

Just noticed that QEMU and raw image builds also followed suit with the same failure reason

Drew Hudson-Viles
2024-10-14 18:00:18

Hmmm, ok. This was tested as working so we'll have to do some digging into it.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-14 18:00:40

I'm just kicking off a CAPV build in my environment to see if I'm seeing the same.

Drew Hudson-Viles
2024-10-14 18:00:42

Sorry I'm phone troubleshooting at the moment but should be back at my pc later to assist where I can.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-14 18:01:14

@Abhay Krishna Arunachalam Do you use the container image?

Abhay Krishna Arunachalam
2024-10-14 18:03:44

no we use a different container image which contains the same versions of components as SCL image-builder has in its ensure-**.sh scripts.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-14 18:04:43

Ah ok, but same version of Packer, etc. yes?

:yes: Abhay Krishna Arunachalam
Abhay Krishna Arunachalam
2024-10-14 18:04:46

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-14 18:07:59
==> vsphere-iso.vsphere: Creating VM...
==> vsphere-iso.vsphere: Customizing hardware...
==> vsphere-iso.vsphere: Mounting ISO images...
==> vsphere-iso.vsphere: Adding configuration parameters...
==> vsphere-iso.vsphere: Set boot order temporary...
==> vsphere-iso.vsphere: Power on VM...
==> vsphere-iso.vsphere: Waiting 3m0s for boot...
==> vsphere-iso.vsphere: Typing boot command...
==> vsphere-iso.vsphere: Waiting for IP...
==> vsphere-iso.vsphere: IP address: 10.10.222.89
==> vsphere-iso.vsphere: Using SSH communicator to connect: 10.10.222.89
==> vsphere-iso.vsphere: Waiting for SSH to become available...
==> vsphere-iso.vsphere: Connected to SSH!
==> vsphere-iso.vsphere: Provisioning with shell script: ./packer/files/flatcar/scripts/bootstrap-flatcar.sh
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-14 18:09:49

I'm not able to reproduce this at least with my setup.
Are you able to share what Make target and what the final combination of user vars and environment varibles are for you're vspehre build (with anything sensitive redacted)

Abhay Krishna Arunachalam
2024-10-14 18:50:46

Running this make target

make -C image-builder/images/capi build-node-ova-vsphere-rhel-8

Christophe Jauffret
2024-10-14 18:57:45

Hello , quickly check and this is expected it is no more working
need to think how we can fix it

Abhay Krishna Arunachalam
2024-10-14 19:03:48

yeah the build before this change passed for us

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-14 19:11:11

Can you check if setting ssh_password to something yourself allows it to build or not?

Abhay Krishna Arunachalam
2024-10-14 19:38:47

kicked off a test build setting the password to a hardcoded string other than builder

Abhay Krishna Arunachalam
2024-10-14 19:40:49

Quick question, won't we also need to change this and other locations since the password is not builder anymore?

GitHub
Christophe Jauffret
2024-10-14 19:50:49

yes

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-14 19:50:50

Oh crap you might be right! I had no idea we hardcoded the password in Ubuntu like that!

Christophe Jauffret
2024-10-14 19:51:03

cloudinit is used to inject password everywhere

Christophe Jauffret
2024-10-14 19:51:12

now we need to templatize the cloud-init content

Abhay Krishna Arunachalam
2024-10-14 19:52:50

also for RHEL in the kickstart file

Abhay Krishna Arunachalam
2024-10-14 19:53:39

but still, these files will be served and the user in the autoinstall will be created only after the SSH communicator goes through and then access the autoinstall files right? So it shouldn't affect the initial SSH connection?
Okay never mind, the boot command is typed before the SSH connection so I think it must definitely look at the builder user being created in the autoinstall file

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-14 20:06:12

@Abhay Krishna Arunachalam what happens if you set the password back to builder in your vars?

Abhay Krishna Arunachalam
2024-10-14 20:24:22

Running a build with builder and a separate one with hello. Will keep the thread posted

👍 Drew Hudson-Viles, Marcus Noble
Abhay Krishna Arunachalam
2024-10-14 20:26:14

I'm curious how it worked in your case though, becuase it seems the SSH password should be the same as what's being created during autoinstall, but builder and the randomly generated UUID would be a mismatch

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-14 20:27:25

I use flatcar which uses ignition rather than the user data

Abhay Krishna Arunachalam
2024-10-14 20:29:20

ah I see. If I kick off a presubmit on image-builder through some dummy change, that would kick off an Ubuntu OVA right?

Abhay Krishna Arunachalam
2024-10-14 20:31:28

In terms of a fix, I think the UUID thing is internal to Packer so it's probably never printed or returned

so I think what we need to do is to generate it outside of Packer (think Makefile or script) and replace in the JSON as well as the user-data

Abhay Krishna Arunachalam
2024-10-14 20:36:58

was the pull-ova-all removed from Prow? I think that might have caught this

Abhay Krishna Arunachalam
2024-10-14 20:44:25

tried this on a linux box

$ uuidgen
04c84fad-871c-4bdf-9307-4ab8f16e5993

Abhay Krishna Arunachalam
2024-10-14 20:49:16

Update: the build with password set to builder passed, but the hello one is stuck waiting for SSH

mboersma
2024-10-14 21:08:18

Yes, pull-ova-all was removed during the community infra migration. We're still waiting for a replacement, see

Abhay Krishna Arunachalam
2024-10-14 21:38:56

Ah I see. Does the fix I suggested above make sense/sound feasible?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 06:01:29

Yeah that sounds feasible to me

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 08:38:51

@Abhay Krishna Arunachalam I just saw your PR! Thank you so much! I’ll take a look at it hopefully later when I’m able to (currently travelling to a conference). Just wanted to check - have you tested if the change works when building a Flatcar image too?

🙌 Abhay Krishna Arunachalam
Abhay Krishna Arunachalam
2024-10-15 08:40:00

so flatcar already has a mkpasswd command wired into a sed command, which i chose not to touch

Drew Hudson-Viles
2024-10-15 08:40:46

You legend, I was about to start looking at the issue this morning! 😄 I'll do some testing on a few things my side and give it my stamp then if all is well from the thing I can test.

💙 Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 08:40:58

🎉 Great! Hopefully I can get it reviewed this morning and get it merged in. I really appreciate the effort!

Abhay Krishna Arunachalam
2024-10-15 08:42:29

Thank you all, appreciate it! I'm also going to test it out on my end since we have a Nutanix/QEMU/Raw/Vsphere testing bed in our CI

Drew Hudson-Viles
2024-10-15 08:43:46

Perfect. I can do the QEMU, Raw and OpenStack to make sure all is working on that side with this change too. (Never hurts to have a couple of perspectives with regards to the QEMU & RAW).

Drew Hudson-Viles
2024-10-15 09:18:55

I'm having a couple issue building QEMU atm, not sure if the envsubst is working as expected in terms of the file it outputs (still contains the $ENCRYPTEDSSHPASSWORD var) - I am flipping between work and this though might be missing something there 😉

Also we should probably consider gitignoring the generated files - i'll stick a note in the issue for that though.

Abhay Krishna Arunachalam
2024-10-15 09:19:57

yeah I had planned to put them in the gitignore too

👍 Drew Hudson-Viles
Abhay Krishna Arunachalam
2024-10-15 09:20:56

Also I might just replace envsubst with sed, since it requires you to have the gettext package installed

Drew Hudson-Viles
2024-10-15 09:35:16

I was wondering if sed might be a better option

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 09:35:27

Either replace with sed or have the binary downloaded into the local bin directory as part of the deps.

Abhay Krishna Arunachalam
2024-10-15 09:37:48

sed works, tested locally

🙌 Drew Hudson-Viles
🎉 Marcus Noble
Christophe Jauffret
2024-10-15 09:41:45

ideal solution would be to migrate the entire image-builder project to packer hcl format so we can use all the built-in function and stop using this kind of hacks 😉
i don’t know if there is plan for that

in all cases the above approach would need some additional modification for platform who are using builder who need to inject userdata directly inside packer config

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 09:44:36

Migrating to HCL is a massive undertaking unfortunately and no one has offered / been able to invest the time into doing it. And as we are sorely lacking in testing its risky to make such a large change to the project as we can't be sure we don't break someones use case. 😞

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 09:45:03

We do want to do it though. It's just not easy 😞

Christophe Jauffret
2024-10-15 09:45:36

yes i totally understood, but for sure one day in the future we will have no more choice

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 09:45:56

Yeah 😞

Drew Hudson-Viles
2024-10-15 09:57:01

I did actually test doing just the OpenStack builder and it's MUCH nice in the HCL format but yeah, huge task to do it for just that one - the whole project would be a lot of work - I'd like to do it but so many edge cases and "hacks" would have to be considered

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 10:07:37

PR looks good to me. I'll leave with @Drew Hudson-Viles to review as he's in a position to test it 🙂 I'll add my lgtm

Abhay Krishna Arunachalam
2024-10-15 10:10:09

Thanks a lot for the quick review!! blod-tada I've put the PR on hold until I have had a chance to test it comprehensively on my end. But I should be able to get it merged today and y'all can then cut a release.

👍 Marcus Noble
Abhay Krishna Arunachalam
2024-10-15 12:15:51

Also realized I need to fix it for Nutanix as well, since currently we have the userdata as a hardcoded base64 string in the packer config file (example), which resolves to

#cloud-config
users:
- name: builder
sudo: ['ALL=(ALL) NOPASSWD:ALL']
shell: /bin/bash
chpasswd:
list: |
builder:builder
expire: False
ssh_pwauth: True

Abhay Krishna Arunachalam
2024-10-15 15:43:54

Fixed Nutanix by adding a static cloud-init template which is base64-encoded and set as the user_data string during build time

Tested it on our CI and it works as expected

Abhay Krishna Arunachalam
2024-10-15 17:20:29

Not sure why the Azure presubmit is failing on the PR, is it unrelated?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 17:21:59

I think that is currently broken and @mboersma is working on removing them from CI.

:ack: Abhay Krishna Arunachalam
Drew Hudson-Viles
2024-10-15 17:23:25

Ftr, I'm still testing things my side. I've been so busy today I've not had much time to go through it all properly I'm afraid. Last time I checked qemu didn't work but haven't synced the branch since this morning.

Abhay Krishna Arunachalam
2024-10-15 17:25:08

i tested nutanix/qemu/raw/vsphere builds on our CI and they all passed

🎉 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 17:26:33

I haven't had a chance to look since this morning and wont now until Thursday. I trust Drew can handle it though 😄 He knows more about these providers than I do

Abhay Krishna Arunachalam
2024-10-15 18:47:47

@mboersma thanks for merging this! Helped get the Azure presubmits passing on my PR

GitHub
👍 mboersma
mboersma
2024-10-15 19:30:18

Good deal! Hopefully we can keep CI greener, it's been kind of flaky.

Abhay Krishna Arunachalam
2024-10-15 20:01:19
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 20:11:19

Look at all that green! 😍

💚 Abhay Krishna Arunachalam, mboersma
Abhay Krishna Arunachalam
2024-10-15 20:17:49

Sorry for the spam! I added my image-builder PR as a patch in eks-anywhere-build-tooling and thought I'd show the CI results from the builds kicked off after the patch merged

GitHub
Abhay Krishna Arunachalam
2024-10-16 16:10:33

@Drew Hudson-Viles thanks for the review! I have addressed your gitignore comment, are you good with that approach?

Drew Hudson-Viles
2024-10-16 16:18:49

Yeah that's absolutely fine. I think I just missed the push where you added it 🙂

Drew Hudson-Viles
2024-10-16 16:19:19

I've focussed on testing each time one comes in rather than reading them tbh. I've done a full read through and review now though so yeah, cracking work buddy!

🙌 Abhay Krishna Arunachalam
Abhay Krishna Arunachalam
2024-10-16 16:19:25

Great! I see you've held it for other reviewers, thanks for the approval!

Drew Hudson-Viles
2024-10-16 16:19:37

Yeah I didn't want to be the only one firing it through 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-16 16:23:59

I should be able to take a look in the morning if no one else gets to it before then. We can then get the released pushed out. 🙂

✅ Drew Hudson-Viles
:ty3: Abhay Krishna Arunachalam
Drew Hudson-Viles
2024-10-16 17:45:46

Thanks for approving that @mboersma! I'll look into getting a release done tomorrow if I can get time in the morning.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-16 17:59:44

I can do a release first thing if you want. Just need the PRs approving. 🙂

👍 Drew Hudson-Viles, mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 07:55:54

Right, starting the release process now 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 08:03:14

😅 seems to be down and causing the build to fail

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 08:03:38

or maybe just really slow 😕

Abhay Krishna Arunachalam
2024-10-17 08:10:52

is the tag creation alone the trigger for the postsubmit?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 08:12:42

Yeah

Abhay Krishna Arunachalam
2024-10-17 08:12:54

In the past I have been able to re-trigger postsubmits by re-delivering the webhook payload corresponding to a PR merge from the repo settings. I'll admit it's not the ideal solution and I'm not sure if there are any side effects because tags are involved

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 08:13:03

oh good idea!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 08:13:29

Oh, Prow isn't configured to use webhooks in the repo 🤦‍♂️

Abhay Krishna Arunachalam
2024-10-17 08:13:36

Ah

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 08:13:47

Yeah, I need someone with more power than me it seems 😅

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 08:15:19

I don't understand why these have been so flakey recently. This is the third release in the past week that has failed initially but I haven't had any problems the previous ~1 year 🤨

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 11:22:19

Looks like someone trigger the re-run for me and its now passed so continuing with the release

🙌 Abhay Krishna Arunachalam
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 11:31:52

@Drew Hudson-Viles

GitHub
✅ Drew Hudson-Viles
Drew Hudson-Viles
2024-10-17 11:35:26

Thanks for taking that on. I'm a 1 man band this week so time is short.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 11:37:14

No worries 🙂 You feeling any better today?

Drew Hudson-Viles
2024-10-17 11:37:45

Yeah much better thanks. Still not 100% but functional non-the-less!

❤️ mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 11:37:58

Oh good! 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 11:43:51

And docs PR:

GitHub
✅ Drew Hudson-Viles
mboersma
2024-10-17 15:52:18

Thanks for doing the release!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 08:30:46

☝️ Following on from this issues (Sorry 😞) I've created an issue to track our progress towards testing as much of the providers and OSs we support in image-builder as possible. Please take a look and let me know if you have any comments or suggestions.

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 08:33:08

/cc @mboersma as I know you've been doing a fair bit of work related to this recently 🙂 Hopefully I haven't missed anything.

:thank_you: mboersma
Christophe Jauffret
2024-10-15 09:49:00

Concerning Nutanix we have an internal daily test on master branch

and the result this morning was not nice thanks to #1596 😰

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-15 10:05:36

That's internal right? Not something we could expose to PRs on image-builder?

Christophe Jauffret
2024-10-15 10:06:14

not for now

:ack: Marcus Noble
leland knight
2024-10-15 23:10:18

I think maybe the packer-plugin-sdk is where the http callback server is being created, I moved the ticket over to that github project:

martin.choe
2024-10-17 04:08:59

@martin.choe has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 11:43:01

Image-builder v0.1.39 is now available: 🎉

Thanks to all contributors! 💙

GitHub
:kubernetes_intensifies: Drew Hudson-Viles, Abhay Krishna Arunachalam, Danny Seymour
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 11:49:31

I'd also like to say a big thank you to @Abhay Krishna Arunachalam for doing so much work to fix the ssh password issue for non-ignition distros! 💙 Huge help and something I was personally stressing about on Monday 😅

🎉 Jacob Weinstock, bavarianbidi
Abhay Krishna Arunachalam
2024-10-17 12:09:25

Happy to contribute, thank you for all the support in reviewing, testing and getting it merged! k8s-heart

karine.santos
2024-10-17 14:58:35

@karine.santos has joined the channel

karine.santos
2024-10-17 15:02:39

Hi everyone! I have a clone of the image builder and need to know the release of this clone. Where can I find this information in the repository?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-17 15:03:43

What do you mean by a clone of image builder? Of the git repo? You should be able to check the git commit to see what your copy is at.

leland knight
2024-10-17 15:46:52

@Marcus Noble or @mcbenjemaa when you get a minute, could you test out my proxmox-related pull request? Looks like the couple folks working on it don't have an environment setup to test with:

Abhay Krishna Arunachalam
2024-10-17 18:57:41

Small fix for the Nutanix gitignore


Sorry missed this in the other PR!

GitHub
:+2: Drew Hudson-Viles
fad3t
2024-10-18 11:27:10

I'm having errors when building on Nutanix. Probably related to the random password thing.
I see the username and password are hardcoded in the user data here, is this expected?

GitHub
fad3t
2024-10-18 11:27:39

and the error I get:

nutanix: output will be in this color.
==> nutanix: Creating Packer Builder virtual machine...
nutanix: Virtual machine ubuntu-2204-kube-v1.30.5 created
nutanix: Found IP for virtual machine: 10.10.141.63
==> nutanix: Using SSH communicator to connect: 10.10.141.63
==> nutanix: Waiting for SSH to become available...
==> nutanix: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
==> nutanix: Force deleting virtual machine...

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-18 11:27:49

Have you updated to v0.1.39?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-18 11:28:22

😞 Damn! i haven't got access to a nutanix environment so not sure I can help. Maybe @Abhay Krishna Arunachalam can assist if available.

fad3t
2024-10-18 11:28:49

I'm having the same issue on vSphere by the way

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-18 11:29:19

Can you share your make target and user vars?

fad3t
2024-10-18 11:29:22

but I'm passing user data as well there

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-18 11:29:33

Ohhhh.... you're overriding the user data?

fad3t
2024-10-18 11:30:03

for vSphere yes, cause the original code uses ubuntu server ISO, while i'm using ubuntu cloud image (just like Nutanix)

fad3t
2024-10-18 11:31:25

I'll have quick lunch then share some more details

👍 Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-18 11:31:26

Oh, I'm not really sure how to handle that. Might need to wait for @Abhay Krishna Arunachalam unless there's someone else here that can help

Christophe Jauffret
2024-10-18 12:01:51

@Marcus Noble with last PR Nutanix build are ok on my side

fad3t
2024-10-18 12:42:29

so my make target is make build-nutanix-ubuntu-2204 but I'm setting PACKERVARFILES with 3 files: .json (Nutanix cluster info), .json (ubuntu22.04 stuff) and .json (K8s version)

fad3t
2024-10-18 12:43:07

I'll see if I can simply remove the user_data from my OS json file, it might have precedence over whats rendered by the template

Christophe Jauffret
2024-10-18 12:44:45

yes remove the user_data from your own file

fad3t
2024-10-18 12:46:12

I confirm it works by removing it from my custom variable file

👍:skin_tone_2: Christophe Jauffret
fad3t
2024-10-18 12:46:29

for vSphere I guess I'll have to maintain a patch file for it

fad3t
2024-10-18 13:26:44

I'm thinking about adding a make target for vSphere + Ubuntu cloud image, I'm just wondering whether people would be interested by this.

Abhay Krishna Arunachalam
2024-10-18 18:45:30

Hey sorry just seeing this. Do y'all still need me to look into something?

fad3t
2024-10-18 20:14:27

Nope, it's fixed - thx

🙌 Abhay Krishna Arunachalam
Brad P
2024-10-19 06:36:47

@Brad P has joined the channel

Brad P
2024-10-19 07:01:53

Hello everyone! I'm new to image builder and trying to get it to build an image using a local vmware workstation 17.6.1 install on Ubuntu 22.04 machine. I cloned the repo and attempted:

 make build-node-ova-local-ubuntu-2204 PACKER_LOG=1
but it always seems to get stuck here:
2024/10/19 05:45:05 packer-plugin-vmware_v1.1.0_x5.0_linux_amd64 plugin: 2024/10/19 05:45:05 [INFO] Attempting SSH connection to 172.16.175.136:22...
2024/10/19 05:45:05 packer-plugin-vmware_v1.1.0_x5.0_linux_amd64 plugin: 2024/10/19 05:45:05 [DEBUG] reconnecting to TCP connection for SSH
2024/10/19 05:45:05 packer-plugin-vmware_v1.1.0_x5.0_linux_amd64 plugin: 2024/10/19 05:45:05 [DEBUG] handshaking with SSH
2024/10/19 05:45:09 packer-plugin-vmware_v1.1.0_x5.0_linux_amd64 plugin: 2024/10/19 05:45:09 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/10/19 05:45:09 packer-plugin-vmware_v1.1.0_x5.0_linux_amd64 plugin: 2024/10/19 05:45:09 [DEBUG] Detected authentication error. Increasing handshake attempts.
I've noticed it seems to be setting a random password each time, and I attempt to login with the password that is set in packer-common.json, for example:
"ssh_password": "VhS9RszwfbP1idLQ",
I get the login prompt, but I can't login myself either, can't figure it out. Any help would be appreciated!

Abhay Krishna Arunachalam
2024-10-19 09:32:35

Which user are you using to login?

Brad P
2024-10-19 18:09:28

builder, it looks to be the user in packer-common.json

Brad P
2024-10-22 03:53:59

Hello, just following-up, this was a pretty basic setup, just minimal install of ubuntu with build essentials, jq, VMware workstation, etc, and the image builder. with all the defaults for the local OVA build, am I missing anything?

Thanks!

Slackbot
2024-10-21 15:30:17

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles
2024-10-21 15:30:43

I'll be on the train to London so won't be able to make this I'm afraid

👍 mboersma
mboersma
2024-10-21 15:30:54

No topics on the agenda currently, but if you have anything please add it!

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-21 15:50:07

I've just arrived at London so can make it if we have anything to discuss 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-21 16:16:35

Agenda still empty. What you wanna do @mboersma?

mboersma
2024-10-21 16:52:46

I guess we skipped! Sorry, was waiting for something to notify that there was an item.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-21 16:54:17

yeah no worries 🙂

hakman
2024-10-25 11:47:57

@hakman has joined the channel

Florian Löhden
2024-10-28 12:49:48

@Florian Löhden has joined the channel

Hani
2024-10-28 15:40:18

@Hani has joined the channel

Hani
2024-10-28 15:47:02

Hello team,
I am having a problem with AWS AMI images build with the latest release of image-builder.
The nodes seem to join the cluster without INTERNAL-IP, therefore calico doesn't run.
Any idea why this might be happening ?

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-28 15:51:05

As far as I'm aware we haven't had any changes recently that might effect that but I can't say for sure without more information.

  • What versions were you running before?

  • What Make target are you using?

  • What user vars are you providing (with anything sensitive removed)?

Hani
2024-10-29 08:14:50

  • I haven't used image-builder before

  • make build-ami-ubuntu-2204

  • these are the only vars i provided

{
"aws_region": "eu-north-1",
"ami_regions": "eu-north-1",
"kubernetes_semver": "v1.30.5",
"ami_groups": "",
"snapshot_groups": "",
"kubernetes_cni_semver": "v1.6.0"
}

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 08:20:33

Oh, so this isn't something thats broken in the latest release but more a case of it not working how you expected? Is that right?

Can you remind me what INTERNAL-IP is in this context? Is that a Kubernetes thing or an EC2 thing?

Hani
2024-10-29 08:23:55

I honestly don't know which case it is exactly.
Basically when you run kubectl get nodes -owide the nodes should have INTERNAL-IP showing up, but in my case it doesn't as if the kubelet doesn't get the address to advertise it

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 08:26:10

Oh gotcha
🤔 Trying to recall where that comes from. I've just checked in one of my own clusters (which uses Flatcar rather than Ubuntu) and I see the IP populated.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 08:26:35

Maybe someone who build Ubuntu AMIs can confirm if they see in the internal IP or not for nodes? 🙏

Hani
2024-10-29 08:31:43

I would add, that i tried building a Flatcar AMI, but there i run into another problem, the kubelet didn't manage to start reporting that no kubelet confing not found
This might be a very unlucky case of things not working for me somehow.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 08:33:45

I just checked my vars from back when we did build ubuntu (months back now) and I noticed that I have this var defined:

"kubernetes_cni_deb_version": "**"
but I don't have kubernetescnisemver. Are you needing that specific version of the cni?

Hani
2024-10-29 08:35:22

No necessarily, i was trying to have the latest versions of network related things, hoping it solves the problem

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 08:37:07

Can you try it without that set and see if that changes things? Would be good to rule it out at least.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 08:37:14

Maybe try setting "kubernetescnideb_version": "**" instead

richcase
2024-10-29 08:49:17

Do you get the same with one of the published AMIs?

Hani
2024-10-29 08:57:03

Yeah, i am having the same problem with published Ubuntu AMIs,

richcase
2024-10-29 08:59:41

CAPA uses calico in its e2e tests for non-EKS clusters.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 09:00:33

Actually, while I think about it - we are talking about non-EKS right?

Hani
2024-10-29 09:18:55

Yeah, non-EKS

Hani
2024-10-29 09:20:59

Building with "kubernetescnideb_version": "" fails the test with this

 {
amazon-ebs.ubuntu-24.04: "duration": 191859,
amazon-ebs.ubuntu-24.04: "err": null,
amazon-ebs.ubuntu-24.04: "expected": [
amazon-ebs.ubuntu-24.04: ""
amazon-ebs.ubuntu-24.04: ],
amazon-ebs.ubuntu-24.04: "found": [
amazon-ebs.ubuntu-24.04: "[\"1.4.0-1.1\"]"
amazon-ebs.ubuntu-24.04: ],
amazon-ebs.ubuntu-24.04: "human": "Expected\n \u003c[]string | len:1, cap:1\u003e: [\"1.4.0-1.1\"]\nTo satisfy at least one of these matchers: [%!s(
matchers.ContainElementMatcher=\u0026{0xc00040b4d0}) %!s(matchers.ContainElementMatcher=\u0026{0xc00040b560})]",
amazon-ebs.ubuntu-24.04: "meta": null,
amazon-ebs.ubuntu-24.04: "property": "version",
amazon-ebs.ubuntu-24.04: "resource-id": "kubernetes-cni",
amazon-ebs.ubuntu-24.04: "resource-type": "Package",
amazon-ebs.ubuntu-24.04: "result": 1,
amazon-ebs.ubuntu-24.04: "successful": false,
amazon-ebs.ubuntu-24.04: "summary-line": "Package: kubernetes-cni: version:\nExpected\n \u003c[]string | len:1, cap:1\u003e: [\"1.4.0-1.1\"]\nTo satisfy at least one of these matchers: [%!s(
matchers.ContainElementMatcher=\u0026{0xc00040b4d0}) %!s(**matchers.ContainElementMatcher=\u0026{0xc00040b560})]",
amazon-ebs.ubuntu-24.04: "test-type": 0,
amazon-ebs.ubuntu-24.04: "title": ""
amazon-ebs.ubuntu-24.04: }

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 09:41:33

Hmmm... maybe that var is no longer valid. As I say this was from months back now 😅

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 09:41:58

But, if the published AMIs are also not working for you I suspect something else is going on instead as I know people are successfully using those images.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 09:42:26

Do you specify any kubeadm config or similar when creating your cluster?

Hani
2024-10-29 09:53:53

nope, just following quickstart guide

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 09:54:58

What quickstart guide? The CAPA one?

Hani
2024-10-29 09:55:22

Yep, that one

Hani
2024-10-29 14:17:00

I can say now, that i managed to pin down the problem to the tigera-operator
thanks for the help

Marcus Noble (k8s@marcusnoble.co.uk)
2024-10-29 14:22:12

Oh interesting. Any idea why?

Hani
2024-10-30 08:27:09

I don't know exactly yet, but this is the error it, coreDNS etc.. give
Error from server: no preferred addresses found; known addresses: []

Hani
2024-10-30 15:56:43

The problem was related to this
which required the CCM to run in host network, i believe i took a long way around to figure out.
thanks for help!

GitHub
timothysc
2024-10-29 21:11:43

@timothysc has left the channel

orin
2024-11-01 10:59:21

Hi all, I have built Flatcar OS image (latest version from main branch on github) and load image into OpenStack Glance, and then I created a server from this image with my custom user data (I show it as yaml file, but I surely converted it to ignition format when booted my server)

variant: flatcar
version: 1.0.0
passwd:
users:
- name: core
password_hash: "$y$j9T$qRgyCaQq.RDwlXNoe.4lS1$srnHt2JI76LZIEQrk1wgMYGvedk/21f0LTWnzH9Z3uB"
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQA..........................xxxxx
- name: stackops
password_hash: "$y$j9T$qRgyCaQq.RDwlXNoe.4lS1$srnHt2JI76LZIEQrk1wgMYGvedk/21f0LTWnzH9Z3uB"
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAA...............................xxxxxx
shell: /bin/bash
groups:
- root
- adm
- wheel
- sudo
- systemd-journal
- docker
storage:
files:
- path: /etc/ssh/sshd_config
overwrite: true
mode: 0600
contents:
inline: |
UsePrivilegeSeparation sandbox
Subsystem sftp internal-sftp
UseDNS no

PermitRootLogin no
AllowUsers core
AuthenticationMethods publickey

But when I tried to login my server using ssh core@x.x.x.x -i the server required me to using password to login, any can explain this bugs to me or give me a solution, I surely that chmod 400 my pem file. Thank you so much

tormath1
2024-11-04 08:44:32

Hi, if you have access to Horizon dashboard you might be able to access the instance console and check that your SSH keys are correctly injected under .ssh/authorizedkeys
Another question: why do you set the password
hash for core user if the goal is to SSH with public keys only? if you enter the password does it work?
EDIT: You can now use this SSHd configuration:

storage:
files:
- path: /etc/ssh/sshd_config.d/custom.conf
overwrite: true
mode: 0600
contents:
inline: |
# Do not allow root user
AllowUsers core

Slackbot
2024-11-04 15:30:18

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2024-11-04 15:31:03

I'm not going to make it unfortunately.

mboersma
2024-11-04 16:58:30

I'm also not available, out of town this week.

hakman
2024-11-04 19:36:31

@hakman has left the channel

leland knight
2024-11-05 03:05:09

I'd like to run image-builder via a gitlab pipeline using a gitlab runner running via kubernetes, but image-builder is using a callback server ... Since I'm running via kubernetes, the local ip of the callback server can't be used, instead I have to expose the ip ... which I am able to do; but I'm not sure how to tell image-builder to use the exposed ip. How can I specify the ip to use?

fad3t
2024-11-05 07:18:15

never found a solution to this, instead I'm using Ubuntu cloud images, which are bootstrapped using cloud-init (no need to get the kickstart config from the Packer web server).

Sriraman Srinivasan
2024-11-05 11:03:36

will httpip , httpportmin and httpport_max do the trick ?

fad3t
2024-11-05 11:12:42

that's difficult, because he says Packer runs in a container so it gets a new (pod) IP every time, and it might not be accessible from outside the cluster (for the VM to reach it)

Sriraman Srinivasan
2024-11-05 11:13:08

Service IP?

fad3t
2024-11-05 11:13:40

what happens if you have 2 parallel runs of image builder?

Sriraman Srinivasan
2024-11-05 11:19:00

Just thinking out loud:
Create onfly service with selector pointing to the specific pod instance?

Sriraman Srinivasan
2024-11-05 11:22:00

Another option is if you use a git based solution, you could host init config files as a server say using nginx.

fad3t
2024-11-05 11:27:18

those config files are built by Packer and are dynamic

Sriraman Srinivasan
2024-11-05 11:30:57

those config files are built by Packer and are dynamic
I am talking about files under say example <a href='https://github.com/kubernetes-sigs/image-builder/tree/main/images/capi/packer/ova/linux'></a>
During the build, packer hosts these files inside http server it creates. Instead of then packer hosting it onfly, you host those seperately

GitHub
fad3t
2024-11-05 11:34:22

I see, this might work yeah

Sriraman Srinivasan
2024-11-05 11:35:42

you could use git based auto-update or something to make sure server is always uptodate

Travis Loyd
2024-11-05 17:04:33

via the gitlab runner helm chart it is possible to ask for a loadbalancer ip, or expose the runner via an ingress FQDN (which gets added to dns automatically), if there were just a place to specify that ... but image-builder tries to use the ClusterIP of the pod which is unreachable.

Travis Loyd
2024-11-05 17:09:26

I was starting to think about submitting a pull request, something like if 'such-and-such' env var existed, use that for the callback ip, otherwise detect the ip automatically ... but it was looking a lot harder than that. ... was hoping it wouldn't require a feature request; seems like this is going to become a more and more common use case

Travis Loyd
2024-11-05 17:11:13

it can still bind to the clusterip, it's just the thing calling back that needs to call the alternative specified fqdn or ip

Travis Loyd
2024-11-05 17:16:24

I'm not sure what is trying use the callback ip, when is the ip getting passed along which is being used to call back on? Its getting passed to the vm somehow to use after things are setup, if I wanted to submit a pull request, which git repo do I need to look at?

Travis Loyd
2024-11-05 17:16:51

we just want to pass an alternative fqdn or ip that we specify via env var

Travis Loyd
2024-11-05 17:37:22

@fad3d i submitted a feature request in the packer project, is that the right place? If so, maybe give it a thumbs up so I'm not alone:

leland knight
2024-11-15 00:48:32

I'm still trying to do this but having trouble figuring out where the code is that I'd have to change. Is the code different for every provider? So if I fixed it for proxmox it would only make proxmox builds work in kubernetes?

leland knight
2024-11-15 00:49:09

Is it in the provider, in imagebuilder, in packer, in the packer-sdk?

Sriraman Srinivasan
2024-11-15 03:20:40

Except the playbooks, majority is provider centric...

leland knight
2024-11-15 03:27:41

How could I influence things such that all providers might one day being able to run via kubernetes? There a kind of "standards group" I could suggest the idea to?

Sriraman Srinivasan
2024-11-15 03:33:34

At this stage of the prj, I guess this is going to be a little difficult since lots of properties specifically wrt packer is provider specific(packer plugins are differnet and hence different set of props required). Plus bootstrap/cloud-init differs again based on provider-OS combination with not all providers supporting all OS flavours....

Sriraman Srinivasan
2024-11-15 03:35:25

May be you can add agenda for this in doc and this then can be taken up during IB office hours

leland knight
2024-11-15 03:36:21

Being that I'm willing to put in the work, I kind of hate to let the idea die, but if it's just not a good idea I can let it go. I can just build images in a way that doesn't run in kubernetes. It's just that I prefer everything run 100% in k8s, making this one the first outlier for me.

👍 Sriraman Srinivasan
leland knight
2024-11-15 03:36:30

Thanks, I'll give that a shot

Travis Loyd
2024-11-05 17:00:17

@Travis Loyd has joined the channel

Travis Loyd
2024-11-05 17:20:12

I'm wanting to specify a different callback ip for the vm to use once it finishes. At what point in the code is the ip being passed to the vm? Is that in the 'image-builder' code?

I submitted a feature request in the packer project, is that the right place?

Gerhard Häring
2024-11-07 05:38:39

@Gerhard Häring has joined the channel

Andrew Sauber
2024-11-08 14:05:21

@Andrew Sauber has joined the channel

karine.santos
2024-11-12 14:41:21

hello everyone!

I use the openstack packer provider in creating my kubernetes images with image-builder. Is there a way for me to see the details of this image (services that go up together, etc.?)

Abhay Krishna Arunachalam
2024-11-12 22:23:56

Do you mean the output image on disk after image-builder has finished building it?

Tobias Thiel
2024-11-14 10:16:03

@Tobias Thiel has joined the channel

feitnomore
2024-11-14 16:29:36

I'm trying to build-qemu-ubuntu-2404, and I'm getting SSH error

feitnomore
2024-11-14 16:30:39

qemu: Setting up proxy adapter for Ansible....
==> qemu: Executing Ansible: ansible-playbook -e packerbuildname="qemu" -e packerbuildertype=qemu -e packerhttpaddr=10.0.2.2:8984 --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerdurl= containerdsha256=041fa3cfd4e6689d37516e4c7752741df0974e7985d97258c1009b20f25f33c7 pauseimage=registry.k8s.io/pause:3.9 containerdadditionalsettings= containerdcrisocket=/var/run/containerd/containerd.sock containerdversion=1.7.20 containerdwasmshimsurl=--linux-x8664.tar.gz containerdwasmshimsversion=v0.11.1 containerdwasmshimssha256={"lunatic":"7054bc882db755ce5f3ded46d114bfd4e0a318e437fa18a2601295d20b616b32","slight":"a6ea87d965037933a7d9edb5e20cfc175265c8e1ca92a16535f1f3c3f376f5b0","spin":"dcffedb8e4d2f585a851b3de489fa1e8a0054ec0ad72cf111c623623919245d0","wws":"e917f90692d798d80873aa0f37990c7d652f2846129d64fecbfd41ffa77799b8"} containerdwasmshimsruntimes="" containerdwasmshimsruntimeversions="{"lunatic":"v1","slight":"v1","spin":"v2","wws":"v1"}" crictlurl= crictlsha256= crictlsourcetype=pkg customrolenames="" firstbootcustomrolespre="" firstbootcustomrolespost="" nodecustomrolespre="" nodecustomrolespost="" disablepublicrepos=false extradebs="qemu-guest-agent" extrarepos="" extrarpms="qemu-guest-agent" httpproxy= httpsproxy= kubeadmtemplate=etc/kubeadm.yml kubernetesapiserverport=6443 kubernetescnihttpsource= kubernetescnihttpchecksum=sha256: kubernetesgoarch=amd64 kuberneteshttpsource= kubernetescontainerregistry=registry.k8s.io kubernetesrpmrepo= kubernetesrpmgpgkey= kubernetesrpmgpgcheck=True kubernetesdebrepo= kubernetesdebgpgkey= kubernetescnidebversion= kubernetescnirpmversion= kubernetescnisemver=v1.2.0 kubernetescnisourcetype=pkg kubernetessemver=v1.29.10 kubernetessourcetype=pkg kubernetesloadadditionalimgs=false kubernetesdebversion=1.29.10-1.1 kubernetesrpmversion=1.29.10 noproxy= pipconffile= pythonpath= redhatepelrpm= epelrpmgpgkey= reenablepublicrepos=true removeextrarepos=false systemdprefix=/usr/lib/systemd sysusrprefix=/usr sysusrlocalprefix=/usr/local loadadditionalcomponents=false additionalregistryimages=false additionalregistryimageslist= ecrcredentialprovider=false additionalurlimages=false additionalurlimageslist= additionalexecutables=false additionalexecutableslist= additionalexecutablesdestinationpath= additionals3=false buildtarget=virt amazonssmagentrpm= enablecontainerdaudit= kubernetesenableautomaticresourcesizing= debugtools=false ubunturepo= ubuntusecurityrepo= gpublocknouveauloading= --extra-vars ansiblepythoninterpreter=/usr/bin/python3 --extra-vars --scp-extra-args "-O" -e ansiblesshprivatekeyfile=/tmp/ansible-key2403733080 -i /tmp/packer-provisioner-ansible683821705 /home/marcelo/local/src/github.com/feitnomore/image-builder/images/capi/ansible/node.yml
qemu:
qemu: PLAY [all] **
==> qemu: ssh: handshake failed: EOF

Slackbot
2024-11-18 15:30:14

Reminder: Image-Builder office hours start in 1 hour. Agenda:

👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-11-18 15:58:54

Not sure I’ll make this. Currently in Copenhagen trying to find my Airbnb 😅

mboersma
2024-11-18 16:13:55

Nice, hopefully that's starting a vacation!

I put a couple things on the agenda just hopefully, but IDK if we actually have any updates. We just haven't met in a while and I wanted to touch base, but if we're too busy today I'll move them to the next slot.

Drew Hudson-Viles
2024-11-18 16:14:37

I won't be around for a few weeks afet today - happy to push if needed but just an fYI there 🙂

I don't have any updates on those items though

mboersma
2024-11-18 16:16:28

Sounds good. We can keep it short if no one has any updates or other topics.

Drew Hudson-Viles
2024-11-18 16:19:50

👍

Marcus Noble (k8s@marcusnoble.co.uk)
2024-11-18 16:24:29

Nice, hopefully that's starting a vacation!
Nope! Speaking at KCD Denmark tomorrow! 😁

👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-11-18 16:25:30

Regarding the two topics on the agenda - there’s no update from my side. Tests are still in the same state I think and the Packer stuff isn’t moving and likely won’t.

mboersma
2024-11-18 16:41:39

Drew and I chatted about topics, but we don't really have any updates yet. I'll carry them over to the next meeting and maybe we've thought of something relevant by then. 🙂

Yongxiang Gao
2024-11-20 03:15:09

I try to use image-builder to build ubuntu 22.04, someone knows how to use USG (Ubuntu Security Guide) to harden the image in image-builder?

I try to use "usg fix cislevel1server" in ansible task, however, when I use the generated osImage to provision EKSA bare metal cluster, I see such error:

"failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"

"failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"

Which tailor rule can be used to skip cilium and kubelet during usg fix?

rajas
2024-11-26 11:08:02

Hey folks! 👋 just created based on the conversation I had with @Marcus Noble at Kubecon!

The idea is to see if there’s appetite for this in the community and work on a proposal together!

GitHub
:parrotk8s: Marcus Noble, Abhay Krishna Arunachalam, Sriraman Srinivasan, galop
❤️ bavarianbidi, chrischdi
👍 tormath1
Marcus Noble (k8s@marcusnoble.co.uk)
2024-11-26 11:17:31

I've also pinned this issue in the repo to try and get more visibility and feedback on it 🙂

:thank_you: rajas, Sriraman Srinivasan
Kepler SysAdmin
2024-11-28 12:35:27

Hi Community,
I've been using the image-builder for about 6-9 months now,
most of the time, it worked correctly, and if I encounter an issue, it was because I was missing something on the var_file.json

Nevertheless, since yesterday, there is something that is quite unique when it tries to build the image.
On the step Add the Kubernetes repo key is failing:

    openstack: TASK [kubernetes : Add the Kubernetes repo key] *
openstack: fatal: [default]: FAILED! => {"after": ["D94AA3F0EFE21092", "871920D1991BC93C"], "before": ["D94AA3F0EFE21092", "871920D1991BC93C"], "changed": true, "fp": "234654DA9A296436", "id": "234654DA9A296436", "key_id": "234654DA9A296436", "msg": "apt-key did not return an error, but failed to add the key (check that the id is correct and *
not
a subkey)", "short_id": "9A296436"}

I think this issue is quite recent and maybe a few people are affected.
The json file I am using:
{
"source_image": "",
"networks": "",
"flavor": "gp1.small",
"floating_ip_network": "public",
"image_name": "ubuntu-2204-kube-v1.27.16",
"image_visibility": "public",
"image_disk_format": "raw",
"volume_size": "20",
"volume_type": "",
"ssh_username": "ubuntu",
"kubernetes_deb_version": "1.27.16-1.1",
"kubernetes_semver": "v1.27.16",
"kubernetes_series": "v1.27"
}

I'll appreciate your help.
Thank you!!!

Kepler SysAdmin
2024-11-28 13:24:18

by the way, such task is:

---
- name: Add the Kubernetes repo key
ansible.builtin.apt_key:
url: "{{ kubernetes_deb_gpg_key }}"
state: present

Where:
"kubernetes_deb_gpg_key": " user kubernetes&#95;series }}/deb/Release.key"
"kubernetes_series": "v1.27"

Hardie Boeve
2024-11-28 17:57:25

@Hardie Boeve has joined the channel

Piyush Maru
2024-11-29 05:52:54

@Piyush Maru has joined the channel

Razvan Mihai
2024-11-29 09:25:29

@Razvan Mihai has joined the channel

Kentaro Abe
2024-12-01 14:28:45

@Kentaro Abe has joined the channel

Slackbot
2024-12-02 15:30:30

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma
2024-12-02 15:38:07

I bumped the topics from last time to this one, but I'm not sure we have any updates. But I'm happy to get together even if it's brief.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-12-02 15:41:02

Yeah no updates as far as I know but it'd be good to meet up as I suspect this'll be that last of the year anyway.

👍 mboersma, rajas
rajas
2024-12-02 16:02:36

I am not able to see this meeting on the kubernetes calendar -

what's the best way to add this meeting to my calendar. Usually joining the mailing list works but that seems to be not working in this case

Google Workspace
mboersma
2024-12-02 16:22:06

Yes, and we're not able to get it on the calendar. It's properly set up in the community repo, but the tooling simply fails to publish several sig-cluster-lifecycle events, and image-builder is one of them.
SCL volunteers have put a fair amount of time into trying to fix this or find a workaround and eventually gave up. Sorry, not a good answer I know.

rajas
2024-12-02 16:25:03

but the context helps. Thanks! I'll setup up a reminder myself for now

👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2024-12-03 11:55:04

Image-builder v0.1.40 is now available:
Thanks to all contributors! 🎉

GitHub
🎉 mboersma, rajas
:thank_you: mboersma, rajas
Kentaro Abe
2024-12-03 16:34:22

Hello community.
I'm trying to build ubuntu-2404-qemu images using VM hosted in Proxmox.
but It couldn't.

first, It takes so long time to connect ssh and those message shown continually. (for about 20 minutes)

2024/12/02 18:45:19 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:19 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/12/02 18:45:19 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:19 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/12/02 18:45:26 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:26 [INFO] Attempting SSH connection to 127.0.0.1:2252...
2024/12/02 18:45:26 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:26 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:45:26 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:26 [DEBUG] handshaking with SSH
2024/12/02 18:45:28 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:28 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/12/02 18:45:28 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:28 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/12/02 18:45:35 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:35 [INFO] Attempting SSH connection to 127.0.0.1:2252...
2024/12/02 18:45:35 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:35 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:45:35 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:35 [DEBUG] handshaking with SSH
2024/12/02 18:45:38 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:38 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/12/02 18:45:38 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:38 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/12/02 18:45:45 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:45 [INFO] Attempting SSH connection to 127.0.0.1:2252...
2024/12/02 18:45:45 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:45 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:45:45 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:45 [DEBUG] handshaking with SSH
2024/12/02 18:45:48 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:48 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/12/02 18:45:48 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:48 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/12/02 18:45:55 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:55 [INFO] Attempting SSH connection to 127.0.0.1:2252...
2024/12/02 18:45:55 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:55 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:45:55 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:55 [DEBUG] handshaking with SSH
2024/12/02 18:45:58 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:58 [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password], no supported methods remain
2024/12/02 18:45:58 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:45:58 [DEBUG] Detected authentication error. Increasing handshake attempts.
2024/12/02 18:46:05 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:46:05 [INFO] Attempting SSH connection to 127.0.0.1:2252...
2024/12/02 18:46:05 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:46:05 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:46:05 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:46:05 [DEBUG] handshaking with SSH

after success to connect, SSH session disconnect unexpectedly.
==> qemu: Provisioning with Ansible...
qemu: Setting up proxy adapter for Ansible....
2024/12/02 18:54:00 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:00 Creating inventory file for Ansible run...
2024/12/02 18:54:00 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:00 SSH proxy: serving on 127.0.0.1:40695
==> qemu: Executing Ansible: ansible-playbook -e packer_build_name="qemu" -e packer_builder_type=qemu -e packer_http_addr=10.0.2.2:8167 --ssh-extra-args '-o IdentitiesOnly=yes' --extra-vars containerd_url= containerd_sha256=041fa3cfd4e6689d37516e4c7752741df0974e7985d97258c1009b20f25f33c7 pause_image=registry.k8s.io/pause:3.9 containerd_additional_settings= containerd_cri_socket=/var/run/containerd/containerd.sock containerd_version=1.7.20 containerd_wasm_shims_url=--linux-x86_64.tar.gz containerd_wasm_shims_version=v0.11.1 containerd_wasm_shims_sha256={"lunatic":"7054bc882db755ce5f3ded46d114bfd4e0a318e437fa18a2601295d20b616b32","slight":"a6ea87d965037933a7d9edb5e20cfc175265c8e1ca92a16535f1f3c3f376f5b0","spin":"dcffedb8e4d2f585a851b3de489fa1e8a0054ec0ad72cf111c623623919245d0","wws":"e917f90692d798d80873aa0f37990c7d652f2846129d64fecbfd41ffa77799b8"} containerd_wasm_shims_runtimes="" containerd_wasm_shims_runtime_versions="{"lunatic":"v1","slight":"v1","spin":"v2","wws":"v1"}" crictl_url= crictl_sha256= crictl_source_type=pkg custom_role_names="" firstboot_custom_roles_pre="" firstboot_custom_roles_post="" node_custom_roles_pre="" node_custom_roles_post="" disable_public_repos=false extra_debs="" extra_repos="" extra_rpms="" http_proxy= https_proxy= kubeadm_template=etc/kubeadm.yml kubernetes_apiserver_port=6443 kubernetes_cni_http_source= kubernetes_cni_http_checksum=sha256: kubernetes_goarch=amd64 kubernetes_http_source= kubernetes_container_registry=registry.k8s.io kubernetes_rpm_repo= kubernetes_rpm_gpg_key= kubernetes_rpm_gpg_check=True kubernetes_deb_repo= kubernetes_deb_gpg_key= kubernetes_cni_deb_version= kubernetes_cni_rpm_version= kubernetes_cni_semver=v1.2.0 kubernetes_cni_source_type=pkg kubernetes_semver=v1.30.5 kubernetes_source_type=pkg kubernetes_load_additional_imgs=false kubernetes_deb_version=1.30.5-1.1 kubernetes_rpm_version=1.30.5 no_proxy= pip_conf_file= python_path= redhat_epel_rpm= epel_rpm_gpg_key= reenable_public_repos=true remove_extra_repos=false systemd_prefix=/usr/lib/systemd sysusr_prefix=/usr sysusrlocal_prefix=/usr/local load_additional_components=false additional_registry_images=false additional_registry_images_list= ecr_credential_provider=false additional_url_images=false additional_url_images_list= additional_executables=false additional_executables_list= additional_executables_destination_path= additional_s3=false build_target=virt amazon_ssm_agent_rpm= enable_containerd_audit= kubernetes_enable_automatic_resource_sizing= debug_tools=false ubuntu_repo= ubuntu_security_repo= gpu_block_nouveau_loading= --extra-vars ansible_python_interpreter=/usr/bin/python3 --extra-vars --scp-extra-args "-O" -e ansible_ssh_private_key_file=/tmp/ansible-key1609096398 -i /tmp/packer-provisioner-ansible138304401 /home/systemadmin/image-builder/images/capi/ansible/node.yml
qemu:
qemu: PLAY [all] **
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 SSH proxy: accepted connection
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 authentication attempt from 127.0.0.1:39488 to 127.0.0.1:40695 as builder using none
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 authentication attempt from 127.0.0.1:39488 to 127.0.0.1:40695 as builder using publickey
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 new env request: LANG=C.UTF-8
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 new exec request: /bin/sh -c '( umask 77 && mkdir -p "echo /tmp/.ansible"&& mkdir "echo /tmp/.ansible/ansible-tmp-1733165641.684557-2704-253976952207637" && echo ansible-tmp-1733165641.684557-2704-253976952207637="echo /tmp/.ansible/ansible-tmp-1733165641.684557-2704-253976952207637" ) && sleep 0'
2024/12/02 18:54:01 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [DEBUG] Opening new ssh session
2024/12/02 18:54:01 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [ERROR] ssh session open error: 'EOF', attempting reconnect
2024/12/02 18:54:01 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [DEBUG] reconnecting to TCP connection for SSH
2024/12/02 18:54:01 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [DEBUG] handshaking with SSH
2024/12/02 18:54:01 [INFO] 0 bytes written for 'stdin'
==> qemu: ssh: handshake failed: EOF
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [INFO] 0 bytes written for 'stdout'
2024/12/02 18:54:01 packer-plugin-ansible_v1.1.2_x5.0_linux_amd64 plugin: 2024/12/02 18:54:01 [INFO] 0 bytes written for 'stderr'
2024/12/02 18:54:01 [INFO] 0 bytes written for 'stderr'
2024/12/02 18:54:01 [INFO] 0 bytes written for 'stdout'
Read from remote host 172.16.223.13: Connection reset by peer
Connection to 172.16.223.13 closed.
client_loop: send disconnect: Broken pipe

Do you have any solutions for this problems?

leland knight
2024-12-08 01:23:40

does the vm have a screen like this?

leland knight
2024-12-08 01:29:34

If so, there should be a line something like this:

2024/12/08 01:27:09 packer-plugin-proxmox_v1.2.1_x5.0_linux_amd64 plugin: 2024/12/08 01:27:09 Found available port: 8395 on IP: 0.0.0.0
==> proxmox-iso.ubuntu-2204: Starting HTTP server on port 8395
Which is an HTTP server setup that the vm calls back to in order to load its configuration. This may be that the vm can't reach back to the HTTP server. To help understand you can test that HTTP server like this (the port may vary):
$ curl localhost:8395

22.04/
24.04/

Kiran
2024-12-04 01:52:34

@Kiran has joined the channel

Kiran
2024-12-04 02:24:29

👋 Hello, team!


I am trying to add a cronjob inside a distroless container that is running the application. It is either giving error or not running correctly. How can get this working as I want to create a cronjob that runs a shell script & a python script.

This is the Dockerfile I'm using:

 # Base image for building the application
FROM
docker.io/debian:12-slim AS build

# Set Python environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Create directory for Gunicorn logs
RUN mkdir -p /app/logs/gunicorn

# Install necessary dependencies and libraries
RUN apt-get update && <br> apt-get install --no-install-suggests --no-install-recommends --yes <br> python3-venv python3-dev default-libmysqlclient-dev build-essential <br> libmariadb-dev pkg-config wget curl gnupg2 unzip cron <br> && apt-get clean <br> && rm -rf /var/lib/apt/lists/**

# Set up Python virtual environment
RUN python3 -m venv /pypi/venv && <br> /pypi/venv/bin/pip install --upgrade pip setuptools wheel

# Copy crontab
COPY web/crontab /etc/cron.d/crontab
RUN chmod 0644 /etc/cron.d/crontab
RUN touch /var/log/cron.log

# Install Python dependencies
FROM build AS build-venv
COPY web/requirements.txt /requirements.txt
RUN /pypi/venv/bin/pip install --disable-pip-version-check -r /requirements.txt pymysql wfastcgi gunicorn gevent

# Final stage: Set up the runtime environment
FROM gcr.io/distroless/python3-debian11

# Copy necessary files from previous stages
COPY --from=build-venv /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu
COPY --from=build-venv /pypi/venv /pypi/venv
COPY --from=build-venv /app/logs /app/logs
COPY --from=build /etc/cron.d/crontab /etc/cron.d/crontab
COPY --from=build /var/log/cron.log /var/log/cron.log

# Set environment variables
ENV PYTHONPATH=web:$PYTHONPATH

# Copy application code
COPY . /app
WORKDIR /app


# Start Gunicorn
ENTRYPOINT ["cron", "&&", "/pypi/venv/bin/gunicorn", "web.APP.wsgi:application", "--bind", "0.0.0.0:8000", "--access-logfile", "/app/logs/gunicorn/access.log", "--error-logfile", "/app/logs/gunicorn/error.log", "--log-level", "info"]


Also tried using a entrypoint.sh script:
 #!/bin/sh

# Start gunicorn in background
/pypi/venv/bin/gunicorn web.APP.wsgi:application --bind 0.0.0.0:8000 --access-logfile /app/logs/gunicorn/access.log --error-logfile /app/logs/gunicorn/error.log --log-level info &

# Run periodic task
while true; do
python3 web/cron_script.py
sleep 300 # Run every 5 minutes
done

But with this also, I don't see the cron_script running.

Marcus Noble (k8s@marcusnoble.co.uk)
2024-12-04 06:18:20

I think you might be asking in the wrong channel. This channel is for the https://github.com/kubernetes-sigs/image-builder project.

GitHub
👍 Sriraman Srinivasan
Abhay Krishna Arunachalam
2024-12-05 08:40:22

Hello image-builder maintainers, I have a fix for some RHEL image build issues which I observed in our CI, and it could potentially happen to anyone installing Ansible collection community.general version >= v10.0.0

GitHub
Sriraman Srinivasan
2024-12-05 09:05:07

@Marcus Noble / @mboersma PR submitted coresponding to issue. If either of you get time, can you please have a look at it.

GitHub
GitHub
Sriraman Srinivasan
2024-12-05 10:27:19

I have marked it as draft... wanted to make a small edit to variablize(not sure if its actually a word) the value for maxsize ...

Sriraman Srinivasan
2024-12-05 19:58:43

@mboersma I have marked it as ready for review and squashed the changeset.

👍 mboersma
Sriraman Srinivasan
2024-12-10 09:15:00

@mboersma Have addressed the comments..

:lgtm: mboersma
Sriraman Srinivasan
2024-12-12 09:45:32

Thanks @mboersma

Sriraman Srinivasan
2024-12-05 09:05:42

Whats the procedure to cut tag/make a new release? What are the conditions we do that(one I am assuming is when CAPI release or K8s release happens)

mboersma
2024-12-05 16:35:21

The release process is described here:

There isn't a release cadence currently, it's more based on when maintainers think changes in main justify tagging it. We just cut v0.1.140 a couple days ago.

leland knight
2024-12-08 01:51:57

Trying to use 'ISO_FILE' w/ proxmox provider:

ISO_FILE="tower:iso/ubuntu-22.04.5-live-server-amd64.iso"
Seeing error:
** one of iso_file, iso_url, or a combination of cd_files and cd_content must be specified for boot_iso

leland knight
2024-12-08 01:52:30

I added it to my proxmox.env file which I'm calling using this script:

$ cat go.sh 
#!/bin/bash

docker run -it --rm --net=host --env-file proxmox.env <br> -v /tmp:/home/imagebuilder/images/capi/downloaded_iso_path <br> registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.40 build-proxmox-ubuntu-2204

Franz
2024-12-10 17:32:48

@Franz has joined the channel

beddari
2024-12-12 09:38:10

@beddari has joined the channel

Rafael Polanco
2024-12-12 21:37:56

@Rafael Polanco has joined the channel

Franz
2024-12-13 13:33:17

Heya, I've been trying to build capi-images for a while now (for all of ubuntu-24.04, ubuntu-22.04, rocky9), but keep getting stuck at the same point -- both with qemu and with the remote-image builder on openstack. In all cases things die with a strange ansible error:

make build-qemu-ubuntu-2404 PACKER_LOG=1
....
2024/12/13 13:43:40 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 [INFO] 0 bytes written for 'stdout'
2024/12/13 13:43:40 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 [INFO] 0 bytes written for 'stderr'
2024/12/13 13:43:40 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 [INFO] RPC client: Communicator ended with: 0
2024/12/13 13:43:40 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 [INFO] 0 bytes written for 'stdin'
qemu:
qemu: TASK [Gathering Facts] *
qemu: fatal: [default]: FAILED! => {"msg": "failed to transfer file to /root/.ansible/tmp/ansible-local-1793709qosm0_m_/tmp5fp2y_ef /tmp/.ansible/ansible-tmp-1734093819.7338624-1793720-280401030368575/AnsiballZ_setup.py:\n\n"}
qemu:
qemu: PLAY RECAP
*
qemu: default : ok=1 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0
qemu:
2024/12/13 13:43:40 packer-plugin-ansible_v1.1.1_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 shutting down the SSH proxy
2024/12/13 13:43:40 [INFO] (telemetry) ending ansible
==> qemu: Provisioning step had errors: Running the cleanup provisioner, if present...
2024/12/13 13:43:40 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 failed to unlock port lockfile: close tcp 127.0.0.1:5998: use of closed network connection
2024/12/13 13:43:40 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 13:43:40 failed to unlock port lockfile: close tcp 127.0.0.1:3020: use of closed network connection
2024/12/13 13:43:40 [INFO] (telemetry) ending qemu
==> Wait completed after 11 minutes 49 seconds
2024/12/13 13:43:40 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2024/12/13 13:43:40 machine readable: qemu,error []string{"Error executing Ansible: Non-zero exit status: exit status 2"}
==> Builds finished but no artifacts were created.
2024/12/13 13:43:40 [INFO] (telemetry) Finalizing.
==> qemu: Deleting output directory...
Build 'qemu' errored after 11 minutes 49 seconds: Error executing Ansible: Non-zero exit status: exit status 2

==> Wait completed after 11 minutes 49 seconds

==> Some builds didn't complete successfully and had errors:
--> qemu: Error executing Ansible: Non-zero exit status: exit status 2
I'd appreciate any pointers on how to fix this.

Drew Hudson-Viles
2024-12-13 13:39:26

are there any other errors that stand out? Unfortunately the failed to unlock port lockfile: close tcp 127.0.0.1:5998: use of closed network connection is usually just a red herring as it's something else that's caused a failure and then this gets output as a result.

I can confirm openstack builds are working with the latest release as I ran one yesterday afternoon that was successful. QEMU should be too but I've not personally tested.

Franz
2024-12-13 13:44:58

I don't see anything else that might cause issues. However, if I run with FOREGROUND=1 I get the following:

make build-qemu-ubuntu-2404 PACKER_LOG=1 FOREGROUND=1
....
==> qemu: Starting VM, booting from CD-ROM
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 Qemu version: 8.2.0
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 Qemu Builder has no floppy files, not attaching a floppy.
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 Executing /root/.local/bin/qemu-system-x86_64: []string{"-display", "gtk", "-vnc", "127.0.0.1:63", "-drive", "if=none,file=output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5,id=drive0,cache=writeback,discard=unmap,format=qcow2", "-drive", "file=/root/.cache/packer/85d1bf86e5e0ecdd6e91515a63cc10bdab146dca.iso,media=cdrom", "-machine", "type=pc,accel=kvm", "-smp", "1", "-cpu", "host", "-device", "virtio-scsi-pci,id=scsi0", "-device", "scsi-hd,bus=scsi0.0,drive=drive0", "-device", "virtio-net,netdev=user.0", "-boot", "once=d", "-netdev", "user,id=user.0,hostfwd=tcp::4092_:22", "-m", "2048M", "-name", "ubuntu-2404-kube-v1.30.5"}
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 Started Qemu. Pid: 1795078
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 Qemu stderr: qemu-system-x86_64: -display gtk: Parameter 'type' does not accept value 'gtk'
==> qemu: Error launching VM: Qemu failed to start. Please run with PACKER_LOG=1 to get more info.
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 failed to unlock port lockfile: close tcp 127.0.0.1:5963: use of closed network connection
2024/12/13 14:41:54 packer-plugin-qemu_v1.1.0_x5.0_linux_amd64 plugin: 2024/12/13 14:41:54 failed to unlock port lockfile: close tcp 127.0.0.1:4092: use of closed network connection
==> qemu: Deleting output directory...
Build 'qemu' errored after 15 seconds 720 milliseconds: Build was halted.

==> Wait completed after 15 seconds 720 milliseconds

==> Some builds didn't complete successfully and had errors:
--> qemu: Build was halted.

==> Builds finished but no artifacts were created.
2024/12/13 14:41:54 [INFO] (telemetry) ending qemu
==> Wait completed after 15 seconds 720 milliseconds
2024/12/13 14:41:54 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2024/12/13 14:41:54 machine readable: qemu,error []string{"Build was halted."}
==> Builds finished but no artifacts were created.
2024/12/13 14:41:54 [INFO] (telemetry) Finalizing.
2024/12/13 14:41:55 waiting for all plugin processes to complete...
2024/12/13 14:41:55 /usr/bin/packer: plugin process exited
2024/12/13 14:41:55 /root/.packer.d/plugins/github.com/YaleUniversity/goss/packer-plugin-goss_v3.2.13_x5.0_linux_amd64: plugin process exited
2024/12/13 14:41:55 /root/.packer.d/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/12/13 14:41:55 /root/.packer.d/plugins/github.com/hashicorp/qemu/packer-plugin-qemu_v1.1.0_x5.0_linux_amd64: plugin process exited
2024/12/13 14:41:55 /usr/bin/packer: plugin process exited
2024/12/13 14:41:55 /root/.packer.d/plugins/github.com/hashicorp/ansible/packer-plugin-ansible_v1.1.1_x5.0_linux_amd64: plugin process exited
2024/12/13 14:41:55 /usr/bin/packer: plugin process exited
2024/12/13 14:41:55 /usr/bin/packer: plugin process exited
make: ** [Makefile:543: build-qemu-ubuntu-2404] Error 1

Franz
2024-12-13 13:46:11

Don't know if that qemu-system-x86_64: -display gtk: Parameter 'type' does not accept value 'gtk' is causing things to die also without FOREGOUND=1 ?

Franz
2024-12-13 13:51:40

For the openstack remote build I also get

PACKER_VAR_FILES=openstack_vars.json make build-openstack-ubuntu-2204
....
openstack:
openstack: PLAY [all] *
openstack:
openstack: TASK [Gathering Facts]

openstack: fatal: [default]: FAILED! => {"msg": "failed to transfer file to /root/.ansible/tmp/ansible-local-1726741dszour5u/tmpj3mgwet2 /tmp/.ansible/ansible-tmp-1733848994.5385716-1726748-172419909510439/AnsiballZ_setup.py:\n\n"}
openstack:
openstack: PLAY RECAP
*
openstack: default : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
openstack:
==> openstack: Provisioning step had errors: Running the cleanup provisioner, if present...
No other errors besides this one.

Drew Hudson-Viles
2024-12-13 14:00:01

ok cool. Well I'm running the qemu one now and it's working fine for me.

  • I presume you're building on linux?

  • Are you using the latest version of image builder?

  • Have you run make deps-xxxx for the appropriate build?

  • With qemu, can you manually launch a qemu instance outside of packer using the binary packer is using? Have a look for the Executing xxxxxx line in the log output when running with PACKERLOG=1 for an example.

  • With OpenStack - does PACKERLOG=1 give any extra information?


They are definitely working though so something locally is causing an issue here -

Franz
2024-12-13 14:57:07

Sorry for being slow in replying -- Christmas activities 🎅 🙂

  • yes linux; it's a VM that runs rocky9

  • I recently pulled the image-builder repo, I'm at commit ace9c5f8f

  • yes I've run both make deps-qemu and make deps-openstack

  • there are actually two executing lines:

  • Executing qemu-img: []string{"create", "-f", "qcow2", "output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5", "20480M"}

  • Executing /root/.local/bin/qemu-system-x8664: []string{"-smp", "1", "-machine", "type=pc,accel=kvm", "-netdev", "user,id=user.0,hostfwd=tcp::3020:22", "-drive", "if=none,file=output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5,id=drive0,cache=writeback,discard=unmap,format=qcow2" "-drive", "file=/root/.cache/packer/85d1bf86e5e0ecdd6e91515a63cc10bdab146dca.iso,media=cdrom", "-boot", "once=d", "-m", "2048M", "-vnc", "127.0.0.1:98", "-cpu", "host", "-device", "virtio-scsi-pci,id=scsi0", "-device", "scsi-hd,bus=scsi0.0,drive=drive0", "-device", "virtio-net,netdev=user.0", "-name", "ubuntu-2404-kube-v1.30.5"}

  • I'm assuming you'd like me to run the latter? Or both?

  • no extra info with OpenstStack and PACKER_LOG=1


I agree, it has to be something with my local environment...

Franz
2024-12-13 15:13:43

I now ran both commands:

qemu-img create -f qcow2 output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5 20480M
Formatting 'output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=21474836480 lazy_refcounts=off refcount_bits=16
and
root/.local/bin/qemu-system-x86_64 -smp 1 -machine type=pc,accel=kvm -netdev user,id=user.0,hostfwd=tcp::3020:22 -drive if=none,file=output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5,id=drive0,cache=writeback,discard=unmap,format=qcow2 -drive file=/root/.cache/packer/85d1bf86e5e0ecdd6e91515a63cc10bdab146dca.iso,media=cdrom -boot once=d -m 2048M -vnc 127.0.0.1:98 -cpu host -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive0 -device virtio-net,netdev=user.0 -name ubuntu-2404-kube-v1.30.5
(diskImageBuilder-venv) [root@cirrus-deploy capi]# /root/.local/bin/qemu-system-x86_64 -smp 1 -machine type=pc,accel=kvm -netdev user,id=user.0,hostfwd=tcp::3020
:22 -drive if=none,file=output/ubuntu-2404-kube-v1.30.5/ubuntu-2404-kube-v1.30.5,id=drive0,cache=writeback,discard=unmap,format=qcow2 -drive file=/root/.cache/packer/85d1bf86e5e0ecdd6e91515a63cc10bdab146dca.iso,media=cdrom -boot once=d -m 2048M -vnc 127.0.0.1:98 -cpu host -device virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive0 -device virtio-net,netdev=user.0 -name ubuntu-2404-kube-v1.30.5
qemu-system-x86_64: warning: Machine type 'pc-i440fx-rhel7.6.0' is deprecated: machine types for previous major releases are deprecated
Now it just sits there. I guess that's a good thing?

Franz
2024-12-13 15:14:10

Gotta run for now. I'll pick this up again on Monday. Thanks a lot for looking into this!!!

Drew Hudson-Viles
2024-12-13 15:17:00

No problem. Yeah once you've run that, it means it's sitting there and the instance should be running. You can use a VNC client to try and connect to ensure it's working. This rules out any issue with the qemu binary you're using anyway if it is working and you can connect.
Have a good weekend and enjoy the festivities!

mboersma
2024-12-16 14:46:17

I have a "soft" conflict with office hours this moring, but I don't see anyting on the agenda, so I'm assuming we will skip. Hopefully I'm not wrong, please follow up here in Slack if there are questions.

Drew Hudson-Viles
2024-12-16 14:47:18

Yes I've got nothing today so I'm happy to skip this one. Not feeling too great either so more rest time is welcomed!

❤️ mboersma
Slackbot
2024-12-16 15:30:09

Reminder: Image-Builder office hours start in 1 hour. Agenda:

LEI
2024-12-16 21:09:04

@LEI has joined the channel

Richard Cunningham
2024-12-22 10:00:11

@Richard Cunningham has joined the channel

Richard Cunningham
2024-12-22 10:02:08

Hi. I am having issues building a qemu Ubuntu 24.04 image (22.04 works ok). The first task for ansible-playbook node.yaml fails to connect SSH. It looks like this issue; . Does anyone have suggestions how to debug further?

GitHub
Slackbot
2024-12-30 15:30:23

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma
2024-12-30 15:37:08

There's nothing currently on the agenda. Let me know if you have anything you'd like to discuss, otherwise we'll skip until next time (13 January 2025).

Sriraman Srinivasan
2024-12-31 13:40:19

New year wished to all and family....

🎉 Drew Hudson-Viles
zakaria
2025-01-03 16:12:38

@zakaria has joined the channel

zakaria
2025-01-03 20:35:48

Hello, we are trying to build a kubernetes VM image from an existing ova template on vSphere, we use the image builder project with the vspher-clone packer builder. At the cloning stage we get the following error knowing that the template folder exist and have in its full path a whitespace. We suspect that the builder does not correctly parse the fullpath ? can anyone help us on this matter? Thank you

Drew Hudson-Viles
2025-01-06 08:30:18

Hi,

I'm not a user of the vSphere side of things so I'm providing complete guesswork from my side.
It does seem like it's expected that no whitespace would be in the path though. The only things I can recommend is to either remove the whitespace from the path if possible or try supplying an escaped value for the space and see if that works.

If not, maybe someone who uses vSphere in anger can supply another option.

👍 zakaria
zakaria
2025-01-06 09:26:57

it seems that the vsphere plugin handles the path join, so we should have only provided the folder name directly not the fullpath 😅.

Drew Hudson-Viles
2025-01-06 09:28:08

Aaah ok! Well glad you sorted it in the end!

👍 zakaria
zakaria
2025-01-06 09:28:43

thank you 😃

Jennifer Weir
2025-01-04 22:07:35

@Jennifer Weir has joined the channel

Benjamin
2025-01-06 08:21:45

@Benjamin has joined the channel

Yongxiang Gao
2025-01-08 05:11:41

Quick question, why there is no preseed-efi.cfg for ubuntu 22.04?
We can see such for ubuntu 20.04 which use base file instead:

./packer/raw/linux/ubuntu/http/base/preseed-efi.cfg
./packer/raw/linux/ubuntu/http/20.04/preseed-efi.cfg

I don't see much packages in packer/raw/linux/ubuntu/http/22.04/user-data, however, the image (tgz file) size is around 2GB, how to reduce it significantly?

Abhay Krishna Arunachalam
2025-01-08 09:36:30

The gz file, when decompressed, yields an image of size 6442450944 bytes which is exactly equal to 6144 MB , the disk size specified here in the raw packer.json. So I think by reducing the size here, you should be able to reduce the final image size. Although I must say I'm not sure about the side effects of doing so.

Yongxiang Gao
2025-01-10 01:51:34

Thanks, I can try, however, I tried to increase that value to 7G for some reason(s) I don't recall.
From a different way, are there packages can be excluded/removed some how to save image size to under 1GB?

s3rj1k
2025-01-13 17:14:49

preseed was deprecated a long time ago, not sure if it even works on newer Ubuntu versions

:ty: Yongxiang Gao
fad3t
2025-01-08 12:57:23

Hi everyone, I'm facing a weird issue and I'm running out of ideas; I'm hoping somebody from this channel would know 😛
I'm building Ubuntu 22.04 images on Nutanix using image-builder, and for some reasons I would like to use a specific "release" of the Ubuntu 22.04 cloud image -- with a specific kernel version/ABI. I've tried setting the image_url to an older release and deployed the resulting template on a cluster, but the kernel version reported doesn't match the one from the Ubuntu release. I've tested with multiple release, they all seem to have kernel version 5.15.0-130-generic which is probably the latest for 22.04.
I'm trying to figure out if the problem comes from the upstream image(s), from the infrastructure building the image or image-builder itself (is it maybe updating the kernel during the build?). Any hint or ideas would be welcome.

Christophe Jauffret
2025-01-08 13:54:23

there is a apt dist-upgrade in the image builder process, i imagine this what replace your kernel version

hajowieland
2025-01-08 21:30:13

@hajowieland has joined the channel

Slackbot
2025-01-13 15:30:29

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-13 15:39:26

☝️ anything to discuss today? Agenda currently empty

mboersma
2025-01-13 16:09:36

Nothing in particular from me. I guess we can punt.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-13 16:12:00

👍 Skipping...

s3rj1k
2025-01-13 16:31:55

hmm, was hoping to jump in 😞

s3rj1k
2025-01-13 16:32:08

ok, so can we do quick offline maybe?

s3rj1k
2025-01-13 16:32:14

hi, all 🙂

mboersma
2025-01-13 16:32:54

I can fire up the zoom meeting just-in-time, no worries.

s3rj1k
2025-01-13 16:33:10

I am there 🙂

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-13 16:52:02

Damn, sorry just seen this. I’m not at my laptop now

mboersma
2025-01-13 16:57:54

No worries @Marcus Noble, we had a good discussion and we'll summarize in the document.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-13 17:04:05

Thanks

s3rj1k
2025-01-13 16:31:21

@s3rj1k has joined the channel

s3rj1k
2025-01-13 17:08:12

Hi all, I was wondering is there someone here that uses raw target to build images? especially Ubuntu and/or Flatcar images

s3rj1k
2025-01-13 17:08:29

cc: @mboersma

👍 mboersma
Abhay Krishna Arunachalam
2025-01-13 22:34:03

Hello, yes we do use image-builder to build Ubuntu 20.04/22.04 and RHEL 8/9 Raw images. Image-builder currently doesn't have Ubuntu 22.04 raw image support so we patch it to include that functionality. The Ubuntu 22.04 subiquity autoinstall files for qemu just worked for raw builds as well and we validate these images in our e2e tests as well

s3rj1k
2025-01-13 23:45:43

Whoa, nice one, please consider doing a pr

mboersma
2025-01-14 16:13:29

There is also this related PR: It would be great if someone could carry that through to merge.

s3rj1k
2025-01-14 19:20:48

^^ I can test raw (Ubuntu) target manually, if some one has the ability to test other changes would be nice, or we can split PR maybe?

s3rj1k
2025-01-14 19:25:29

asked original author in PR about this

s3rj1k
2025-01-15 19:58:21

I've created a consolidated PR for this,
Please take a look

s3rj1k
2025-01-16 11:01:08

@mboersma ^^ can you take a look please?

s3rj1k
2025-01-18 11:44:06

tests seem to pass

s3rj1k
2025-01-23 15:19:11

so are we able to land that change?

zakaria
2025-01-20 08:43:20

Hello all, I am using the build-node-ova-vsphere-clone-ubuntu-2204 to build a kubernetes v1.30.7 vSphere ova from the base template cluster-api-provider-vsphere 1.30.0. I am trying to build it without exposoing an HTTP server to serve the meta-data and user-data for the cloud-init, instead I tweeked the packer-node configuration for the vsphere-clone builder to use the cdfiles and cdlabel in order to pass them to the VM in an An iso CD. Unfortunatly, the build process get stuck on the Waiting fot ssh to be available step with the following msg, I suspect that the cloud-init does not take into consideration the user-data file? Any input on this issue would be appriciated.

zakaria
2025-01-20 08:46:09

I have added the following to the user-data config apt:
preservesourceslist: false
primary:
- arches: [default]
uri:

chrischdi
2025-01-20 14:05:34

I’d take a look to the vm packer creates maybe you get some clue out of that.

zakaria
2025-01-21 08:18:26

when I connect to the VM using the vSphere web console, I get the login prompt but no creds work for me, I have tried to connect with the builder user and its corresponding password (that I have changed in the set-ssh-paasword script so it does not generate one randomly)

zakaria
2025-01-21 08:19:05

i tried login: ubuntu and passwd ubuntu also it does not work

chrischdi
2025-01-21 09:09:14

Maybe you get some hint when watching the screen while it comes up

Sriraman Srinivasan
2025-01-22 06:08:42

@zakaria Depending on the commit you are running from, if older, builder/builder should work if I am not mistaken. In case of newer commit, check the console(script) to get the password(password get generated randomly during start and gets logged).

Sriraman Srinivasan
2025-01-22 06:13:48

Also if you are able to see that the VM has booted correctly with login prompt, check if the IP assigned to the VM and the one packer is waiting on are the same. If different, increase the ipsettletimeout to say 20m or something depending upon the the time taken to boot up of the VM. Sometimes when bootup takes time (usually due to slower network - primarily to connect to Ubuntu servers to do update), in such cases DHCP re-assigns a new IP and packer will not know of the new IP and waits on the older IP. ipsettletimeout will cause packer to wait for the mentioned time before trying to get the IP of the VM

zakaria
2025-01-22 08:16:32

@Sriraman Srinivasan I am using a static IP, with the linux customize options ( It seems that image builder project does not support it by default, I had to add the customize config to the packer-node.json file), and I set the ssh_host to the same ip address.

zakaria
2025-01-22 08:18:56

I am using the latest image builder version v0.1.40 that uses the packer version v1.9.5

zakaria
2025-01-22 08:22:15

I have hardcoded the $ENCRYPTEDSSHPASSWORD to have a simple password in the set-ssh-password script, As I understand in the user-data config, it sources the builder user password from this variable.

zakaria
2025-01-22 08:22:35

but it does not work 😅

zakaria
2025-01-22 08:24:00

I suspect that the cloud-init does not take into consideration the user-data I have configured

zakaria
2025-01-22 08:25:33

I have set the user-data as a VM property, also does not work

SM
2025-01-20 16:27:44

@SM has joined the channel

leland knight
2025-01-23 22:06:41

A humble request:

Though we all use image-builder to create the images for our kubernetes clusters we are currently unable to use it within kubernetes pipeline solutions. So, if you use an on-prem solution such a gitlab or jenkins, running in kubernetes, you will not be able to run image-builder.

This is because the hashicorp packer project has a limitation which breaks via kubernetes. I've created a fix, which requires three tiny changes in three git repos. I humbly request, if you would like to run image-builder within a kubernetes environment that you give my pull request over there a thumbs up. I'm not sure the folks reviewing my pull request have enough kubernetes experience to really understand how freeing using their tool via kubernetes would be.



Thank you for your consideration.

Additionally, I began working on a strategy to test everything image-builder related, but my solutions are always kubernetes based meaning I can't pursue those goals without being able to run things within a kubernetes environment. ()

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 06:51:15

Why are you unable to run in Kubernetes exactly? I run image-builder in Kubernetes with Tekton pipelines successfully for CAPA, CAPZ and CAPV

fad3t
2025-01-24 07:33:36

I'm also running image-builder within Kubernetes, with CAPX and CAPV.
For CAPV I have to apply a patch to the Makefile because we're using Ubuntu cloud images (so vApps) but overall its OK.

leland knight
2025-01-24 15:29:46

because when the vm is complete it sends an http message back to packer, the ip it uses is the ip of the pod, the clusterip, and so the vm fails to signal that its finished

leland knight
2025-01-24 15:30:34

if you've already got a solution for that i'm all ears, i've put a ton of effort into this and would love to know there's already a method to make it work

fad3t
2025-01-24 15:32:01

If i'm not mistaken the HTTP call is used to get the "kickstart" (not sure about the term here). So if you're using cloud-init, this is not needed as all the information needed to bootstrap the VM is already known (via userdata).

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 15:33:39

What make target are you calling?

leland knight
2025-01-24 15:34:42

i'm essentially running this same command but wanting to do it via kubernetes rather than on my vm with docker:

docker run -it --rm --net=host --env-file proxmox.env <br>  -v /tmp:/home/imagebuilder/images/capi/downloaded_iso_path <br>  registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.38 build-proxmox-ubuntu-2204
()

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 15:36:58

🤔 Maybe this is something Proxmox-specific then?

leland knight
2025-01-24 15:38:27

I have tried to solve the issue in a way that would be easy for any provider to implement. Logically, each provider could implement their own unique non-standard work around, which I suspect the more popular providers have probably done.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 15:41:05

Do you know what exactly in the VM performs the callback? I didn't think Packer actually ran on the VMs when building but I could be wrong about that.

leland knight
2025-01-24 15:43:36

there's a place in the proxmox provider where it binds to 0.0.0.0 the http server the vm will talk with, it then runs a function that tries to get the ip address of the system, then provides the ip address it finds to the vm; my solution adds a new variable to the packer sdk, then the proxmox provider checks if that variable exists, if it does it uses that as the ip address instead of running the function to get the ip of the system id did the 0.0.0.0 bind with; there is another variable which looks it can be used for that purpose but its slightly different in that it offers an alterative bind address when it exists instead of 0.0.0.0

fad3t
2025-01-24 15:46:16

it this the HTTP server mentioned here then?

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 15:48:12

Ah ok, so this is a Proxmox feature. That explains why no one else has reported it. We have very few Proxmox users.
It's this functionality, right?

GitHub
leland knight
2025-01-24 15:50:31

I don't think its necessarily proxmox specific; Its been awhile, but when I was deep diving this a few months ago I think what I found was some sample code that people could use when they wanted to put together a provider, and that sample code has this problem (feature?) in it. So I'd say its more sample-code related, and probably in multiple providers, than specifically proxmox.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 15:51:04

Sorry, let me say "proxmox-specific in image-builder"

leland knight
2025-01-24 15:51:08

though at the moment i'd have to dive in again, i'm not sure if it was clusterapi sample code or image builder sample code

fad3t
2025-01-24 15:52:18

i'm not sure this is specific to Proxmox, is it? when using ubuntu 22 live server ISO, it expects the boot command, which usually refers to an HTTP server (which is hosted by the Packer host)

leland knight
2025-01-24 15:53:19

one sec, reviewing my own pull requests which point out where it is in the code

leland knight
2025-01-24 15:54:28

here's the proxmox specific change, but i suspect the same if statement is in all the providers, i believe this is sample code ... i'll look for the sample code so i can show you what i mean ...

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 15:54:29

Ok I guess what I really want to know then is why this isn't impacting the other providers in the same way then. If we can figure that out then maybe we can solve it for Proxmos

fad3t
2025-01-24 15:55:51

for CAPV I end up with the same issue if I'm using the Ubuntu live server ISO

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 15:56:30

Oh really?!

fad3t
2025-01-24 15:56:35

because the packer pod has a non routable (pod) IP, which is set by Packer in the boot command - but the VM cannot reach it

👍 leland knight
fad3t
2025-01-24 15:57:04

might work if using a routable pod IP, never had a chance to test this

leland knight
2025-01-24 15:57:50

I suspect if we ran image-builder via kubernetes with every provider and each of the images many would build the image, timeout after like 20 minutes, and fail because the vm couldn't report in.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 15:57:56

ok - so the reason I've never hit this in our environment then is we build Flatcar not Ubuntu which I guess doesn't have this behaviour because of ignition.

👍 leland knight
Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 15:58:48

But how are we not having lots of people complaining about this? 😕 Surely this would be a blocker for plenty of people?

fad3t
2025-01-24 15:59:09

I remember seeing a couple of issues from people failing to build Ubuntu on CAPV

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 15:59:38

OH! Is that this same issue?

leland knight
2025-01-24 16:01:13

maybe when it doesn't work people just give up and switch to another solution, or they aren't running pipelines in k8s cause they just haven't got to that point in their experience with kubernetes, or they just use aws and maybe that one works (i haven't tried), or when it doesn't work they just give up and run docker via a vm and hope someone else fixes the problem some day

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:01:18

This issue?

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:01:44

You might be right 😞

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:02:55

I'm guessing then that ssh doesn't become available because the VM didn't initialise and setup ssh server because it couldn't reach back to packer

👍 leland knight
leland knight
2025-01-24 16:03:13

that looks like it, "not planned", yet my pull request would solve that one ... so i should link it to my pull request

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:04:01

"not planned" is because it went stale as no-one could solve it or offer more info

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:04:15

(I hate stale bots)

leland knight
2025-01-24 16:04:15

which is understandable, i worked on it for weeks before figuring it out

fad3t
2025-01-24 16:04:19

so your PR would allow to override the HTTP server IP right? how would you determine the IP to use then?

leland knight
2025-01-24 16:05:05

it took a long time cause i had to learn code in three repos to solve it, and well open source so didn't have a ton of time ... but i became a little obsessed i'll admit

😆 Marcus Noble
leland knight
2025-01-24 16:05:27

@fad3t we'd just have to set it via an environment variable

leland knight
2025-01-24 16:05:51

in my case i use gitlab which uses a gitlab-runner in kubernetes and it makes the ip available as an env var

fad3t
2025-01-24 16:06:35

but what IP would you set? cause the runner has a dynamic IP, every time it runs it gets a different IP

leland knight
2025-01-24 16:07:32

exactly

leland knight
2025-01-24 16:07:59

the helm chart has a setting 'use loadbalancer' or just clusterip, so i set that to loadbalancer

fad3t
2025-01-24 16:08:14

I see, thx

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:08:38

Hold on, before we go too far with this - we have tests in place on image-builder that successfully build ubuntu-2204 on Azure. Maybe we should have a look at what that provider does differently.

leland knight
2025-01-24 16:10:15

sorry, i am supposed to be at work ... so i have to come back to this when my day ends, i can look into that log, but i can look into their custom solution after work, and can work on things this weekend

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:10:53

Oh I'm not saying you need to solve it, or we need to solve it right now. I'm just saying its something to look at as an alternative possible solution 🙂

leland knight
2025-01-24 16:11:06

i'm definitely motivated to get this implemented, and my preference is to solve it for all providers if possible, rather than rewrite proxmox and let the issue remain elsewhere

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:11:45

Sure, but if we can solve it without needing a load balancer then that would be my personal preference 🙂

leland knight
2025-01-24 16:11:58

since you both understand the issue, if you could help to communicate it over in that packer-sdk pull request, that'd be helpful to help out the person making the decision to include it or not

leland knight
2025-01-24 16:12:56

@Marcus Noble i suppose there's nothing stopping us from putting in the kubernetes specific url, in my case something like gitlab-runner.gitlab.svc so we can use the clusterip

leland knight
2025-01-24 16:13:19

but we'd still need the implementation i've put together (as far as i can tell)

leland knight
2025-01-24 16:14:05

maybe if you could both just add a note over there that the issue does exist, and that the pull requests could solve it, though i understand folks might decide as a group there's a better way

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:14:07

What I mean is the Azure provider manages without the need of calling back to a http server it seems. I'd like to understand how it does that.

leland knight
2025-01-24 16:15:04

maybe that could be an action item for you?

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:15:09

Ah! I think Azure uses Azure Resource Manager which gives it more capabilities

👍 leland knight
Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:15:14

Yeah, I'm looking

👍 leland knight
Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:16:20

My concern mainly is even if we get that PR in, how do we handle this in environments where the cluster isn't routable from the cloud (that's the case for me). Also, what security implications are there in opening up that endpoint to the web? (As I don't understand this enough)

leland knight
2025-01-24 16:17:22

it could just be like pod.cluster.svc

leland knight
2025-01-24 16:17:30

it doesn't have to be a loadbalancer

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:17:35

We also have GCP in our tests building Ubuntu20.04 -

leland knight
2025-01-24 16:17:42

i'd use an ingress for sure, but it'd all just be internal in the end

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:18:35

Yeah, that'd work in your specific case but I'm trying to think about the other environments this may run. And also thinking about the recent security incidents we've had and make sure we don't introduce anything more. 🙂

leland knight
2025-01-24 16:18:51

dang it, i want to work on this, but have to go to actual job ... afk for awhile, will be in and out

👍 Marcus Noble
leland knight
2025-01-24 16:18:56

ya, let's solve it for all providers

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:19:01

Just to be clear - I'm not trying to block anything here, I just want to figure out all the options 🙂

leland knight
2025-01-24 16:19:12

so everyone isn't implementing something unique, unique solutions make fore security vulnerabilities

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:19:31

Yeah, but I suspect that is totally what's happening 😞

leland knight
2025-01-24 16:19:37

me too

fad3t
2025-01-24 16:19:37

Azure and GCE seem to use pre-existing images (kind of AMIs) so they probably don't need the boot command

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:20:05

Ah! Good point! I guess that'll be the same for AWS too then.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:22:44

I need to head off also as it's end of my day now. I'm going to try and think through this over the weekend. If we go down this route I think we need some automated way to expose the endpoint securely but I'm not sure we will be able to as there's nothing that says image-builder is running in a cluster. 🤔

fad3t
2025-01-24 16:23:57

at the end its just a way to override the IP address that is announced to the VM, its not changing the way it is actually running

fad3t
2025-01-24 16:24:58

if there's a security risk linked to this, then it is already present as packer is already running on that image builder k8s pod

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-24 16:25:02

Yeah, I’m just trying to figure out how this will work in image-builder

Drew Hudson-Viles
2025-01-24 21:44:17

I've just read all of this thinking "but I don't have this problem in OpenStack either" - then I read the last 4 comments and the penny dropped. The problem here is systems that don't use the boot command are generally fine. Anything that does evidently seems to be having this issue.

That's a fascinating bug though and whilst it appears to be an edge case where it errors out, it's not an unlikely scenario that would cause it IE someone wanting to actually build something in K8S which relies on a boot command.

Not sure what the solution is yet, just wrapping my head around it all myself and thought I'd chime in with my 2 cents!

👍 leland knight, Marcus Noble, mboersma
chrischdi
2025-01-26 10:33:01

For vsphere/capv afaik it also works in a mode where the cloud init data is provided via an iso instead of relying on the http callback.

chrischdi
2025-01-26 10:39:20

Because upstream ci (when it still existed) did run packer in a pod inside the prow cluster. There was no way to call back from the VM to the pod where packer was running.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 09:08:13

Any idea how that is used @chrischdi? Looks like we install it into the container image so maybe we can use that for Proxmox then? I dunno what it actually does though.

chrischdi
2025-01-27 09:09:28

One question is: does the proxmox plugin allow to mount a second iso to get the cloud-init data from.

chrischdi
2025-01-27 09:09:48

and allow to upload the built iso (so it can be mounted). But I never used proxmox.

chrischdi
2025-01-27 09:10:02

(I guess it allows to do so but can’t confirm)

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 09:10:40

You can upload ISO to proxmox, that bit I know. I don't know about the first. Can you show me how it's used in vsphere provider and I'll see if I can figure it out

chrischdi
2025-01-27 09:11:56

I think its a default packer feature.
For OVA builds if I’m right these are the important lines for it:

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 09:13:11

Ah cool, and then its used like here:

chrischdi
2025-01-27 09:13:39

ack yeah that makes sense

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 09:13:53

ok, let me investigate then 🙂

chrischdi
2025-01-27 09:14:06

(the cd_label is also important for this if I remember correctly)

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 09:14:19

cidata ?

chrischdi
2025-01-27 09:14:28

jep

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 09:15:11

🤔

chrischdi
2025-01-27 09:15:15

even documented for proxmox:

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 09:17:37

ah yes! It's about half way down the page (annoyingly no anchor to link to)

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 09:19:49

So it might be possible to add:

  "cd_content_location": "./packer/proxmox/linux/{{user distro&#95;name}}/http/{{user distro&#95;version}}/**",
"cd_label": "cidata",
to the proxmox ubuntu values?

chrischdi
2025-01-27 09:22:59

could be it, yeah

chrischdi
2025-01-27 09:23:45

IMHO the most robust way for this stuff as the network back to packer is not needed.

:nod: Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 09:24:12

Until this thread I didn't even know the http call was how it worked 🙈

chrischdi
2025-01-27 09:39:06

😄 well its not how it always works 🙂

chrischdi
2025-01-27 09:39:18

but afaik is the default when using packer

leland knight
2025-02-13 21:14:46

If no one has come up with a better way, maybe consider the solution I've put together and comment on the pull requests / github issues.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-02-14 06:55:32

Have you tried the cd approach? That would make it inline with others and not require HTTP connectivity.

leland knight
2025-02-14 17:38:04

I'm not sure what you mean by cd, so no probably. I'll scroll up in this chat and look for 'cd'.

leland knight
2025-02-14 17:41:44

Still, I mean ... even if there is a workaround, why not solve it so a work-around isn't required.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-02-14 18:09:53

It isn’t a workaround. It’s the approach used elsewhere in the project.

leland knight
2025-02-14 18:10:31

I feel resistance to the idea I've proposed, I don't want to go against the flow, could you help me to see why the idea I've suggested isn't a good idea?

leland knight
2025-02-14 18:11:02

I'm just curious, genuinely interested in your insight before I give up ... I mean, the hard part has already been done ... I've already implemented a solution.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-02-14 18:26:48

I’m not suggesting that. I’m just asking to try the approach that we already have used with other providers and see if that solves the problem. That way we don’t need any external changes.

leland knight
2025-02-14 18:28:47

k, sorry, what do you mean by cd? that translates as 'continous delivery' in my brain

leland knight
2025-02-14 18:32:37

this cd?

Marcus Noble (k8s@marcusnoble.co.uk)
2025-02-14 19:35:14

Yeah that. Sorry I’m on my phone. I should have got you a link.

👍 leland knight
leland knight
2025-02-14 19:35:58

no worries, working on several other things at the moment ... appreciate the response

👍 Marcus Noble
Victor Sartori
2025-01-27 11:44:41

@Victor Sartori has joined the channel

Slackbot
2025-01-27 15:30:03

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 15:31:32

☝️ Anyone want to use the office hours to discuss the above thread? (Or is the thread good enough for now?) Any other topics as the agenda is currently empty?

Drew Hudson-Viles
2025-01-27 15:41:28

I'm easy on this - I think the thread is ok for the moment unless we've some immediate action we want to take?

Drew Hudson-Viles
2025-01-27 15:42:00

I do have to leave to pick my daughter up at 4:45 though so I'd have to do a quick one for me 😄

mboersma
2025-01-27 15:44:54

I don't have any agenda items in particular, but happy to talk. I just now caught up with the megathread.

mboersma
2025-01-27 15:45:17

(Was out on Friday last week)

Drew Hudson-Viles
2025-01-27 15:46:12

tbf, It has been a while so at the very least we should have a catch up 🙂

👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 15:46:42

Works for me 🙂

s3rj1k
2025-01-27 16:07:56

I have a question in regards to PR, so it would be nice to have some short meetup

Marcus Noble (k8s@marcusnoble.co.uk)
2025-01-27 16:11:28

I'm having some internet issues so I might not actually be able to join. Go ahead without me if I'm not there

👍 Drew Hudson-Viles
Andy Townsend
2025-01-28 20:46:30

@Andy Townsend has left the channel

Ilya Alekseyev
2025-02-09 23:08:58

@Ilya Alekseyev has joined the channel

laozc
2025-02-10 07:41:06

@laozc has joined the channel

Danila B.
2025-02-10 13:27:24

@Danila B. has joined the channel

Slackbot
2025-02-10 15:30:02

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles
2025-02-10 15:30:51

Unfortunately I won't be able to make this one. We've a new starter today and I'm in meetings until around 5:30 UK time.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-02-10 15:32:04

I'm also not going to make it today but the agenda is looking empty so I think it's ok to skip.

👍 Drew Hudson-Viles
mboersma
2025-02-10 15:33:04

Works for me, see you next time!

icelynjennings
2025-02-18 20:36:25

@icelynjennings has joined the channel

Vitaliy Kozlovskiy
2025-02-19 18:19:28

@Vitaliy Kozlovskiy has joined the channel

Amulyam24
2025-02-20 06:47:10

@Amulyam24 has joined the channel

Priyanshi Khetwani
2025-02-20 06:47:21

@Priyanshi Khetwani has joined the channel

Priyanshi Khetwani
2025-02-20 07:08:04

Hi Team,
I am trying to build an image using below command

PACKER_LOG=1 PACKER_FLAGS="--var 'kubernetes_rpm_version=1.30.4-0' --var 'kubernetes_semver=v1.30.4' --var 'kubernetes_series=v1.30' --var 'kubernetes_deb_version=1.30.4-00'" make build-qemu-ubuntu-2204
but getting ssh handshake error
Attempting SSH connection to 127.0.0.1:2877...
reconnecting to TCP connection for SSH
handshaking with SSH
SSH handshake err: ssh: handshake failed: read tcp 127.0.0.1:33368->127.0.0.1:2877: read: connection reset by peer

Also if I try without some packer variables, i am able to build the image
PACKER_LOG=1 PACKER_FLAGS="--var 'kubernetes_semver=v1.30.4'" make build-qemu-ubuntu-2204
Can someone please help?

cc: @Amulyam24

Marcus Noble (k8s@marcusnoble.co.uk)
2025-02-20 15:35:22

I think you might be hitting this issue (which closed without a resolution by the looks of it 😞 )

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2025-02-20 15:35:48

Does anyone have a process for adding custom tags to AWS AMIs when building them with image-builder?

Slackbot
2025-02-24 15:30:09

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2025-02-24 15:31:32

I had totally forgot about this 🤦‍♂️ I'm going to need to skip. Got too much that still needs to get done.

mboersma
2025-02-24 15:33:00

No worries Marcus! I can be there but the agenda is currently empty, so unless someone speaks up soon I think we will skip until next time.

:thx_thanks: Marcus Noble
Abhay Krishna Arunachalam
2025-02-24 18:49:07

Hello @Marcus Noble @mboersma @Drew Hudson-Viles just curious, what is the tentative date for the next IB release v0.1.41?

mboersma
2025-02-24 18:51:36

The project doesn't have a release schedule, we've been doing releases when "enough" new features or bug fixes have landed. We may be overdue for one.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-02-24 18:54:09

Ah! I knew there was something I wanted to mention 🤦‍♂️ Yeah. I think we're overdue one, I noticed we have a few unreleased merges the other day.

mboersma
2025-02-24 18:54:33

There's a PR in the queue right now that may merge, but after that I can do a release.

💙 Marcus Noble, Abhay Krishna Arunachalam
👍 Drew Hudson-Viles
Abhay Krishna Arunachalam
2025-02-24 20:40:08

Thank you!

mboersma
2025-02-24 21:42:37

Thank you for the nudge!

🙌 Abhay Krishna Arunachalam
Abhay Krishna Arunachalam
2025-02-24 21:43:54

Especially excited for this one because a couple of IB patches we're maintaining on EKS-anywhere side have been upstreamed (thanks to @s3rj1k 🎉)

❤️ mboersma, s3rj1k, Marcus Noble
mboersma
2025-02-25 15:52:48

kubernetes Image-builder v0.1.41 is now available:
Thanks to all contributors!

🙌 Drew Hudson-Viles, Victor Sartori, Abhay Krishna Arunachalam, s3rj1k
mohamed karim
2025-02-25 16:36:28

@mohamed karim has joined the channel

mkumatag
2025-02-26 00:37:58

@mkumatag has left the channel

Mo
2025-02-27 02:05:42

@Mo has joined the channel

Razvan Mihai
2025-02-27 09:57:40

@Razvan Mihai has left the channel

Ayan Khan
2025-03-03 11:14:59

@Ayan Khan has joined the channel

Ayan Khan
2025-03-03 11:56:29

How can we know the password of the default user "ubuntu". Or is there any option to configure it as I'm not able to find anything related to this. Found in a discussion that a random UUID is taken for password during image build process.

Drew Hudson-Viles
2025-03-03 11:59:37

It depends on the provider you're using.

For example, the QEMU provider has it dynamically set via a script which generates a packer.json from this template:

Other builders will allow you to pass it in as a var - you'll have to look at which provider you are using and the builder it invokes to pass the correct value through.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-03 12:00:03

The UUID password is for the "builder" user that should only be used during the initial image creation. I thought you're asking for an ssh user within the generated image at the end, yes?

Drew Hudson-Viles
2025-03-03 12:00:41

aaah yeah sorry, I may have misread that Marcus - good catch about the resulting image 😄

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-03 12:01:27

I don't thing, by default, we generate any ssh user in the resulting image. (I could be wrong, not sure about all providers and OS's)

Drew Hudson-Viles
2025-03-03 12:02:22

To my knowledge no we do not. The method I mentioned is about setting one during the build process.

Generally speaking you would pass in things like credentials/user creation or public keys for your ssh keypair via cloud-init .

:gr_correct: Sriraman Srinivasan
Ayan Khan
2025-03-03 12:16:25

Thanks everyone! Really appreciate the help.

Dave Miles
2025-03-05 10:55:11

@Dave Miles has joined the channel

Victor Sartori
2025-03-05 16:42:09

Hey everyone,

I recently opened a PR to add support for a new provider, Canonical MaaS.
What’s the process for adding a new provider? Should this be discussed with someone in particular, or would it be a good topic for the next office hours meeting?
Here’s the PR:

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-06 11:24:08

Hey Victor, I'm sorry I meant to respond to your PR and just lost track of time with everything going on.
I'm having a chat with the other maintainers about this (and related) as we're at a tipping point where we're not sure if we can commit to managing any more providers or operating systems. We think we might need to come up with some sort of minimum set of requirements for adding new ones but not sure yet what that might be. Ideally, any new providers would also come with tests but that's really only viable if the providers themself are willing to donate infrastructure to run tests on (do you know if this is something Canonical is likely to be interested in?)
I think it makes sense to talk about this in the office hours.

/cc @mboersma @jsturtevant @Drew Hudson-Viles

Victor Sartori
2025-03-06 11:31:59

Hello Marcus,
I'm not sure if Canonical is interested in this. By the way, I don't work at Canonical but at another company that is integrating with MaaS.
Regarding donating infrastructure to run MaaS, I can check internally at my company. It might be feasible, but I need to discuss it with many people to make it happen.
IMHO, I think this would be a very interesting and important step for the CAPI project to support bare metal directly. As you know, with the rise of AI and related technologies, the use of bare metal will increase considerably, and MaaS makes it much easier to deploy a Kubernetes cluster.

Victor Sartori
2025-03-06 11:33:37

The next office hour meeting will be Mon, 10th march?

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-06 11:37:08

Correct

Victor Sartori
2025-03-06 11:37:51

and can I add this topic to the agenda?

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-06 11:40:52

Yeah go right ahead 🙂

Victor Sartori
2025-03-06 11:41:02

🙂

Victor Sartori
2025-03-14 11:02:59

Hello @Marcus Noble
Sorry to bother you with this, but I'd like to go over some points we discussed in the last office hours.
I left a comment on the PR , asking if my understanding of our discussion was correct.
I'm not sure if I fully understood which README (or if this should perhaps go into the book?) and what its content should be. If you could help with some topics, I can continue from there.
I'm asking because I want to speed up the process of getting this provider merged into master as much as possible, so we can move forward with our project more smoothly.

GitHub
Victor Sartori
2025-03-14 11:05:55

And regarding the owners file, who is responsible for updating it? Should I do it? How does this process work?

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-14 11:08:48

I’m currently travelling, @Drew Hudson-Viles or @mboersma are you available to help out?

Victor Sartori
2025-03-14 11:09:56

Oh, Marcus, I'm really sorry to bother you with this!! Enjoy your trip!

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-14 11:11:27

It’s totally fine 🙂

Drew Hudson-Viles
2025-03-14 11:49:51

We're just launching a (quiet) go live today so might have limited time but I will try and find some if I can and if Matt doesn't get in there before me!

Victor Sartori
2025-03-14 11:51:28

Got it! no worries if it’s not possible today. Good luck with the go-live!

👍 Drew Hudson-Viles
Yongxiang Gao
2025-03-06 09:36:22

Does image-builder support full disk encryption (boot disk) for ubuntu 22.04 now?

Drew Hudson-Viles
2025-03-06 21:13:30

Hi, responded in the DM you sent 🙂

:ty: Yongxiang Gao
hemaa
2025-03-08 13:30:25

@hemaa has joined the channel

Thomas Güttler
2025-03-10 08:45:08

Why chronyd, when provider has matching NTP set?

I compare the result of make build-hcloud-ubuntu-2404 with a vanilla ubuntu24.04 created in hcloud.

The vanilla image has this setting:

/etc/systemd/timesyncd.conf.d/hetzner.conf:NTP=ntp.hetzner.com

I think this works fine.

Why does image-builder install chronyd?

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-10 08:52:56

Looks like originally it was added for Azure I think?


Not sure if it's a case of copy from one provider to another.

Thomas Güttler
2025-03-10 12:38:08

thank you for the reply. I see that there is a lot of chrony config in image-builder. I will see if I can disable it.

chrischdi
2025-03-10 12:38:29

no background on ntp in image-builder, from personal experience in the past: chronyd behaves way better and I/we had experience in the past when using timesyncd

mboersma
2025-03-10 15:17:17

Apparently our automated reminder expired, but image-builder office hours start in 15 minutes. Agenda:

👍 Drew Hudson-Viles
:thx_thanks: Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-10 15:20:50

Daylight savings time change 😄

mboersma
2025-03-10 15:21:03

D'oh of course.

mboersma
2025-03-10 15:22:02

So the time switched here, but not yet in the UK? So is ~10 minutes from now the right time, or am I just making US-based assumptions?

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-10 15:22:24

The UK switched ages ago. We're back on UTC 😄

mboersma
2025-03-10 15:22:29

Ah

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-10 15:22:39

My notification popped up so I think you're right

mboersma
2025-03-10 15:22:53

ok

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-10 15:22:54

As we schedule it based on US time

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-10 15:23:49

Maybe I should delete this and let you recreate? Then it should (hopefully) maintain the right time with the US?

mboersma
2025-03-10 15:24:40

Sure, we can try that, maybe that's the way.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-10 15:24:55

Deleted 🙂

mboersma
2025-03-10 15:29:36

Ok, I recreated it for 8:30 am my time. Had to fiddle around to get it to start two weeks from now.

💙 Marcus Noble
mboersma
2025-03-10 15:26:48

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 9:30AM every other Monday (next occurrence is March 17th), Mountain Daylight Time.

mboersma
2025-03-10 15:27:44

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 9:30AM every other Monday (next occurrence is today), Mountain Daylight Time.

mboersma
2025-03-10 15:29:11

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/1YIOD0Nnid_0h6rKlDxcbfJaoIRNO6mQd9Or5vKRNxaU/edit” in this channel at 8:30AM every other Monday (next occurrence is March 24th), Mountain Daylight Time.

Slackbot
2025-03-10 15:30:16

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Abhay Krishna Arunachalam
2025-03-11 06:13:15

Hello image-builder maintainers, I was trying to build GPU-ready OVAs by setting the vsphere ISO builder's pcipassthroughalloweddevice field. vSphere allows users to assign multiple PCI passthrough devices (GPU cards, video capture cards, audio cards, etc) to a virtual machine without specifying an exact physical device on a particular ESXi host. It does this through a feature called Dynamic DirectPath I/O, which requires virtual hardware version 17. However since we're hardcoding this to 15, I'm not able to get this feature working. This version hasn't been changed in 5 years, I think we should update this outdated version or remove it. From the Packer documentation, the vsphere builder's vmversion field, if not set, defaults to the most current virtual machine hardware version supported by the ESXi host. Kindly let me know your thoughts on this.

Abhay Krishna Arunachalam
2025-03-11 06:52:10

We have overriden the hardware version to 18 for windows 2019 and windows 2022, but only because the corresponding guest OS types were not supported in hardware version 15. Maybe we should bump the default to a newer version while still allowing overriding for backward compatibilty with older vCenter versions?

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-11 06:58:22

It sounds like the unset default might actually be best if I understand correctly. That would then always use the latest version available in the environment you’re building the image, yes?
We should still allow it to be set if needed for backward compatibility though.

Abhay Krishna Arunachalam
2025-03-11 07:01:58

I haven't tried the unset default route myself and for the time being, am resorting to overriding vmx_version in the image-specific packer file (rhel-8 OVA.json for example), but based on the documentation, it seems like it should work

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-11 07:04:55

If theres no objections from others I'd be in favour of that. But I don't have all that much knowledge of OVA so maybe worth someone else weighing in first.

chrischdi
2025-03-11 20:48:43

In case you are using capv: you should be able to overwrite the hardware version: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/blob/main/apis/v1beta1/types.go#L205

GitHub
Cristian Vlad
2025-03-11 14:37:50

@Cristian Vlad has joined the channel

Dragos Nicu
2025-03-11 15:02:10

@Dragos Nicu has joined the channel

jawnsy
2025-03-18 23:49:22

@jawnsy has joined the channel

Victor Sartori
2025-03-19 17:11:43

Hello everyone!

I came across a need that I couldn't find in Ansible for the Image Builder.
I need to set some kernel parameters for the OS via sysctl. These parameters must persist after a reboot (and should also prevent users from having to define them through init containers, for example).

What would be the most viable solution for this?
Should I create a new role that copies the parameters to /etc/sysctl.d/98-custom.conf?
Or would it be better to have a generic role that copies any local file to a specified path in the image? This way, we could also solve future issues related to custom files that need to be added to the image.

Depending on our discussion here, I can work on this task

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-19 17:33:33

My suggestion would be to use nodecustomroles_post and supply your own custom ansible role.

👍 Sriraman Srinivasan
Victor Sartori
2025-03-19 17:39:49

Got it Marcus!! Thanks for the tip!

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-19 17:40:56

If it turns out there are others interested in similar then we can port it upstream into image-builder but as of right now I haven’t seen anyone looking for similar that wasn’t very specific so might be best with the custom role. 🙂

Victor Sartori
2025-03-19 17:43:21

Very fair. I had a feeling that something to solve this kind of problem had already been considered.

Anand Kumar
2025-03-20 18:42:28

@Anand Kumar has joined the channel

Slackbot
2025-03-24 14:30:04

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2025-03-24 14:32:10

I'll be skipping today. I have a bunch of things I'd like to get ready for KubeCon next week 😅

👍 mboersma
Drew Hudson-Viles
2025-03-24 16:31:30

It looks like there isn't anything in the agenda anyway - shall we skip altogether?

mboersma
2025-03-24 18:20:58

Done

Jan
2025-03-24 19:22:50

@Jan has joined the channel

Abhay Krishna Arunachalam
2025-04-02 22:27:09

Hello Image-builder maintainers, when trying to build RHEL 9 OVAs in CI, we ran into this error for all Kubernetes versions

Build 'vsphere-iso.vsphere' errored after 3 minutes 29 seconds: No host is compatible with the virtual machine.
After some digging, I discovered this is because the RHEL 9 Guest OS type rhel9_64Guest was introduced in vSphere API release 7.0.1.0 (source), which is compatible with virtual machine hardware version 18 (source). But image-builder hardcodes the VM hardware version to 15, which doesn't support RHEL 9. We worked around this on our end by patching image-builder after which the builds succeeded. I'm upstreaming the patch in this PR. Kindly take a look and let me know if there are any concerns. Thank you!

GitHub
👍 mboersma
Abhay Krishna Arunachalam
2025-04-02 22:28:58

I'm also curious if others ran into this, I'd be surprised if they didn't thinking

Abhay Krishna Arunachalam
2025-04-03 19:20:36

cc @mboersma @Drew Hudson-Viles @Marcus Noble

Marcus Noble (k8s@marcusnoble.co.uk)
2025-04-03 19:35:21

check-success
Sorry for the delay. Currently at KubeCon and busy busy busy 😆

Abhay Krishna Arunachalam
2025-04-03 19:56:32

Ah sorry for the untimely ping! Really appreciate you reviewing and merging the PR despite your busy schedule! ty

Marcus Noble (k8s@marcusnoble.co.uk)
2025-04-03 20:01:35

Nah don't worry about it 🙂 Always appreciate the contributions!

🙌 Abhay Krishna Arunachalam
Raj Surve
2025-04-04 02:03:13

@Raj Surve has joined the channel

oivindoh
2025-04-04 11:19:38

@oivindoh has joined the channel

Slackbot
2025-04-07 15:30:05

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma
2025-04-07 16:21:53

Nothing on the agenda, but I'm happy to have office hours if anyone has anything to talk about. Speak up in the next few minutes if so... :-)

mboersma
2025-04-07 16:31:53

Ok, let's skip today.

Please add to the agenda for next time if you have something to discuss or present, or ask your questions here in the Slack channel. 😄

Drew Hudson-Viles
2025-04-07 17:03:24

Sorry, I've been stuck in meetings and didn't even see the pop-ups for slack 🤦‍♂️

Drew Hudson-Viles
2025-04-07 17:03:45

Brains still recovering from 12.5k people over 5 days 🤣🤣

:kubernetes: mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2025-04-08 05:15:51

:face_palm: sorry, I was out yesterday for my friends birthday and totally forgot shot this.

Thomas Güttler
2025-04-11 18:04:08

@Thomas Güttler has left the channel

shameemshah
2025-04-18 10:54:04

@shameemshah has joined the channel

Slackbot
2025-04-21 15:30:02

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2025-04-21 15:32:24

Anyone else no longer have access to the agenda? Or is it just me for some reason?

Marcus Noble (k8s@marcusnoble.co.uk)
2025-04-21 15:32:51

Also, if possible I’d rather we skip today but can make it if needed

Drew Hudson-Viles
2025-04-21 15:59:03

Yeah I'm being prompted to request access. But yes, bank holiday weekend in the UK, so won't be available myself.

mboersma
2025-04-21 16:04:11

I have the same problem--I also lost access to another SCL Google Doc at the same time.

mboersma
2025-04-21 16:05:06

But I have access to CAPI and CAPZ docs, so it's not a matter of me not having the right perms. I think we need the actual document owner to refresh its permissions so Kubernetes in general can see it.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-04-21 16:12:16

Do you know who that is? 😅

mboersma
2025-04-21 16:13:24

I don't--maybe @jsturtevant does? Or hopefully someone in sig-cluster-lifecycle leadership, I can ask Fabrizio.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-04-22 08:13:11

I had a tab still open with the agenda to I managed to grab a backup of it before reloading.
You can view it here:
It's currently set to comment-only as I don't want this to become the new agenda as it's tied to Giant Swarm but I wanted to make sure we didn't lose the history 🙂

👍 Drew Hudson-Viles, mboersma
mboersma
2025-04-23 16:11:14

I think Fabrizio is an owner, he said he could help us move it to the appropriate k8s document area where apparently it should have been to avoid this problem. Crossing fingers.

👍 Drew Hudson-Viles
mboersma
2025-04-23 16:58:33

@Marcus Noble we couldn't find a new owner but Fabrizio created a new doc here:

Drew Hudson-Viles
2025-04-23 17:18:13

But... how will I know unless I can see the doc 😄

Ahmet Beyazoğlu
2025-04-22 09:13:20

@Ahmet Beyazoğlu has joined the channel

Ahmet Beyazoğlu
2025-04-22 10:45:35

Hello everyone,
We’ve encountered an issue while using kubernetes-sigs/image-builder to build Red Hat-based node images for Cluster API (CAPI) workload clusters.
After provisioning a workload cluster with these images, we’ve noticed that /etc/resolv.conf on the nodes includes two unexpected nameserver entries . These are not defined in our bootstrap data, cloud-init config, or Image Builder templates.
Example output of resolve.conf:

; Created by cloud-init automatically, do not edit.
Generated by NetworkManager
search foo.bar local
nameserver 10.x.x.1
nameserver 10.x.x.2

These additional nameservers are not configured by the user and their origin is unknown.

Has anyone seen similar behavior or could point us to where these values might be coming from during the build/provisioning process? (I think these nameservers come from DHCP scope during CI build, but they should not be in the template.)

Buid : build-node-ova-vsphere-rhel-8 (including with "packer-node.json" )
Rhel ISO: rhel-8.10-x86_64
Image builder version : cluster-node-image-builder-amd64:v0.1.41

Happy to share more details if needed. Thanks in advance!

Tomas Dabašinskas
2025-04-25 06:40:00

@Tomas Dabašinskas has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2025-04-29 11:32:28

Image-builder v0.1.42 is now available:

Thanks to all contributors! 💙

GitHub
🎉 Drew Hudson-Viles, mboersma
:kubernetes_intensifies: Drew Hudson-Viles, mboersma
Slackbot
2025-05-05 15:30:08

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Drew Hudson-Viles
2025-05-05 15:31:14

Another bank holiday in the UK and I'm away so I'm not going to be around for this I'm afraid. Sorry!

Marcus Noble (k8s@marcusnoble.co.uk)
2025-05-05 15:42:57

Same! ☝️

Marcus Noble (k8s@marcusnoble.co.uk)
2025-05-05 15:43:43

But it also seems like the agenda doc isn’t sorted yet(?) so no items anyway I guess

Arnaud Pons
2025-05-05 16:18:48

Hello all,

I've made a feature request regarding the support of ARM64 azure VM template build:


I've described the list of detected blockers based on what I've experienced on my side.
As said in the issue I'm not confortable to make a PR for this due to the lack of knowing on how you would like to see this feature handled.
Trying to add a single dedicated makefile target (i.e. sig ubuntu 22.04 ARM64) seems to cause a lot of duplicated code, and on the other hand, trying to add the notion of processor architecture more globaly needs to make a lot of modifications on things I don't know enough and I'm not able to test.

I'm open to feedback and will be happy to help where I can to see this feature supported 🙂.

Software Engineer
2025-05-15 16:38:34

@Software Engineer has joined the channel

Gokul Girish
2025-05-16 04:36:31

@Gokul Girish has joined the channel

Gokul Girish
2025-05-16 06:33:26

Hello Everyone,
I'm understanding that Ubuntu Latest Versions (22.04 and 24.04) do not supports preseeds anymore, so it's not referenced over here .

How can I configure the preseed - to add changes to cloudinit / auto-install file ? I tried updating base/preseed-efi.cfg.tmpl and 22.04.efi user-data.tmpl but I don't see it has reflected. Also in the logs on trying to build make build-raw-ubuntu-2204-efi I can see multiple preseeds and user-data (cloud inits) being logged. What's the order of precedence

Could someone please guide on this ?
TIA

GitHub
Sriraman Srinivasan
2025-05-16 14:11:32

@Marcus Noble / @mboersma Can you please take a look at PR. This adds ability for user to specify ansible roles to be run post the sysprep stage. Currently there are no hooks which allow user to run custom playbooks/roles after all the stages(including sysprep but before goss validation runs).
cc: @rajas

GitHub
👍 Gokul Girish
Sriraman Srinivasan
2025-05-20 05:07:37

@Marcus Noble / @mboersma Can you please have a look at the above PR?

Marcus Noble (k8s@marcusnoble.co.uk)
2025-05-20 05:09:24

Both me and Matt are currently away this week. Is it ok if we get to it next week when we’re back?

Sriraman Srinivasan
2025-05-20 05:21:14

@Marcus Noble Let me see if I can get someone else to look at these. Please do not bother your break. Sorry for pinging during your break(didn't realize that).

Marcus Noble (k8s@marcusnoble.co.uk)
2025-05-20 05:57:10

It’s totally fine. I’m actually away for work so I might get chance to look at it for you but it’s not guaranteed

🙏 Sriraman Srinivasan
Sriraman Srinivasan
2025-05-21 07:40:05

@chrischdi Can you please have a look at the PR ?

chrischdi
2025-05-21 18:52:56

(Did see this notification but not sure when I get back to it)

👍 Sriraman Srinivasan
Sriraman Srinivasan
2025-05-27 05:22:26

@Marcus Noble / @mboersma Once you are back, can you please have a look at the PR?

Marcus Noble (k8s@marcusnoble.co.uk)
2025-05-27 06:47:55

👍 I'm going to try and get to it today but I have a lot to catch up on so apologies if it takes me a while

👍 Sriraman Srinivasan
Marcus Noble (k8s@marcusnoble.co.uk)
2025-05-27 08:59:38

LGTM. Assigned to the others for approval but if no one comes back by this afternoon I'm happy to merge it 🙂

:ty_thankyou: Sriraman Srinivasan
mboersma
2025-05-27 16:17:18

Sorry, all, I'm back now. Thanks for being on top of things @Marcus Noble!

💙 Marcus Noble, Sriraman Srinivasan
mboersma
2025-05-16 15:51:35

Image-builder v0.1.43 is now available:

Thanks to all contributors! heart-kube

🎉 Sriraman Srinivasan
Slackbot
2025-05-19 15:30:04

Reminder: Image-Builder office hours start in 1 hour. Agenda:

mboersma
2025-05-19 16:20:11

We’ll have to skip today’s office hours, as the active maintainers are either on break or at an offsite.

Karine Santos
2025-05-20 21:38:34

@Karine Santos has joined the channel

Karine Santos
2025-05-21 00:39:11

[Packer Build Failing with "ssh: handshake failed: EOF" on Ubuntu 24.04]

Karine Santos
2025-05-21 00:42:14

Hi everyone,
I’m encountering an issue during a Packer build where the Ansible provisioner fails with an SSH handshake error. The build consistently crashes at the same step, and I’d appreciate any insights.
Environment

  • OS: Ubuntu 24.04 (LTS)

  • Packer version: 1.9.4 (QEMU builder)

  • Error:

==> qemu: ssh: handshake failed: EOF  
2025/05/20 23:11:46 [ERROR] ssh session open error: 'EOF', attempting reconnect
Current Configuration
I’ve already adjusted SSH settings for resilience:
{
"builders": [
{
"accelerator": "{{user accelerator}}",
"boot_command": [
"{{user boot&#95;command&#95;prefix}}",
"{{user boot&#95;media&#95;path}}",
"{{user boot&#95;command&#95;suffix}}"
],
"boot_wait": "{{user boot&#95;wait}}",
"cd_files": [
"{{user cd&#95;files}}"
],
"cd_label": "cidata",
"cpu_model": "host",
"cpus": "{{user cpus}}",
"disk_compression": "{{ user disk&#95;compression}}",
"disk_discard": "{{user disk&#95;discard}}",
"disk_image": "{{ user disk&#95;image }}",
"disk_interface": "virtio-scsi",
"disk_size": "{{user disk&#95;size}}",
"firmware": "{{user firmware}}",
"format": "{{user format}}",
"headless": "{{user headless}}",
"http_directory": "{{user http&#95;directory}}",
"iso_checksum": "{{user iso&#95;checksum&#95;type}}:{{user iso&#95;checksum}}",
"iso_url": "{{user iso&#95;url}}",
"memory": "{{user memory}}",
"net_device": "virtio-net",
"output_directory": "{{user output&#95;directory}}",
"qemu_binary": "{{user qemu&#95;binary}}",
"shutdown_command": "echo '{{user ssh&#95;password}}' | sudo -S -E sh -c 'usermod -L {{user ssh&#95;username}} && {{user shutdown&#95;command}}'",
"ssh_password": "{{user ssh&#95;password}}",
"ssh_timeout": "2h",
"ssh_username": "{{user ssh&#95;username}}",
"type": "qemu",
"vm_name": "{{user vm&#95;name}}",
"vnc_bind_address": "{{user vnc&#95;bind&#95;address}}"
}
],
"post-processors": [
{
"environment_vars": [
"CUSTOM_POST_PROCESSOR={{user custom&#95;post&#95;processor}}"
],
"inline": [
"if [ \"$CUSTOM_POST_PROCESSOR\" != \"true\" ]; then exit 0; fi",
"{{user custom&#95;post&#95;processor&#95;command}}"
],
"name": "custom-post-processor",
"type": "shell-local"
},
{
"environment_vars": [
"OUTPUT_DIR={{user output&#95;directory}}",
"ARTIFACT_NAME={{user artifact&#95;name}}",
"KUBEVIRT={{user kubevirt}}"
],
"inline": [
"if [ \"$KUBEVIRT\" != \"true\" ]; then",
"exit 0",
"else",
"bash ./packer/qemu/scripts/build_kubevirt_image.sh {{user build&#95;name}}-container-disk",
"fi"
],
"name": "kubevirt",
"type": "shell-local"
}
],
"provisioners": [
{
"environment_vars": [
"PYPY_HTTP_SOURCE={{user pypy&#95;http&#95;source}}"
],
"execute_command": "BUILD_NAME={{user build&#95;name}}; if [[ \"${BUILD_NAME}\" == \"flatcar\" ]]; then sudo {{.Vars}} -S -E bash '{{.Path}}'; fi",
"script": "./packer/files/flatcar/scripts/bootstrap-flatcar.sh",
"type": "shell"
},
{
"ansible_env_vars": [
"ANSIBLE_SSH_ARGS='{{user existing&#95;ansible&#95;ssh&#95;args}} {{user ansible&#95;common&#95;ssh&#95;args}}'",
"KUBEVIRT={{user kubevirt}}"
],
"extra_arguments": [
"--extra-vars",
"{{user ansible&#95;common&#95;vars}}",
"--extra-vars",
"{{user ansible&#95;extra&#95;vars}}",
"--extra-vars",
"{{user ansible&#95;user&#95;vars}}",
"--scp-extra-args",
"{{user ansible&#95;scp&#95;extra&#95;args}}"
],
"playbook_file": "./ansible/firstboot.yml",
"type": "ansible",
"user": "builder"
},
{
"expect_disconnect": true,
"inline": [
"sudo reboot now"
],
"inline_shebang": "/bin/bash -e",
"type": "shell"
},
{
"ansible_env_vars": [
"ANSIBLE_SSH_ARGS='{{user existing&#95;ansible&#95;ssh&#95;args}} {{user ansible&#95;common&#95;ssh&#95;args}}'",
"KUBEVIRT={{user kubevirt}}"
],
"extra_arguments": [
"--extra-vars",
"{{user ansible&#95;common&#95;vars}}",
"--extra-vars",
"{{user ansible&#95;extra&#95;vars}}",
"--extra-vars",
"{{user ansible&#95;user&#95;vars}}",
"--scp-extra-args",
"{{user ansible&#95;scp&#95;extra&#95;args}}"
],
"playbook_file": "./ansible/node.yml",
"type": "ansible",
"user": "builder"
},
{
"arch": "{{user goss&#95;arch}}",
"format": "{{user goss&#95;format}}",
"format_options": "{{user goss&#95;format&#95;options}}",
"goss_file": "{{user goss&#95;entry&#95;file}}",
"inspect": "{{user goss&#95;inspect&#95;mode}}",
"tests": [
"{{user goss&#95;tests&#95;dir}}"
],
"type": "goss",
"url": "{{user goss&#95;url}}",
"use_sudo": true,
"vars_file": "{{user goss&#95;vars&#95;file}}",
"vars_inline": {
"ARCH": "amd64",
"OS": "{{user distro&#95;name | lower}}",
"OS_VERSION": "{{user distribution&#95;version | lower}}",
"PROVIDER": "qemu",
"containerd_version": "{{user containerd&#95;version}}",
"kubernetes_cni_deb_version": "{{ user kubernetes&#95;cni&#95;deb&#95;version }}",
"kubernetes_cni_rpm_version": "{{ split (user kubernetes&#95;cni&#95;rpm&#95;version) \"-\" 0 }}",
"kubernetes_cni_source_type": "{{user kubernetes&#95;cni&#95;source&#95;type}}",
"kubernetes_cni_version": "{{user kubernetes&#95;cni&#95;semver | replace \"v\" \"\" 1}}",
"kubernetes_deb_version": "{{ user kubernetes&#95;deb&#95;version }}",
"kubernetes_rpm_version": "{{ split (user kubernetes&#95;rpm&#95;version) \"-\" 0 }}",
"kubernetes_source_type": "{{user kubernetes&#95;source&#95;type}}",
"kubernetes_version": "{{user kubernetes&#95;semver | replace \"v\" \"\" 1}}"
},
"version": "{{user goss&#95;version}}"
}
],
"variables": {
"accelerator": "kvm",
"ansible_common_vars": "",
"ansible_extra_vars": "ansible_python_interpreter=/usr/bin/python3",
"ansible_user_vars": "",
"artifact_name": "{{user build&#95;name}}-kube-{{user kubernetes&#95;semver}}",
"boot_media_path": "http://{{ .HTTPIP }}:{{ .HTTPPort }}",
"boot_wait": "10s",
"build_timestamp": "{{timestamp}}",
"cd_files": "linux/base/**.nothing",
"containerd_sha256": null,
"containerd_url": " containerd&#95;version}}/cri-containerd-cni-{{user containerd&#95;version}}-linux-amd64.tar.gz",
"containerd_version": null,
"cpus": "1",
"crictl_url": " crictl&#95;version}}/crictl-v{{user crictl&#95;version}}-linux-amd64.tar.gz",
"crictl_version": null,
"disk_compression": "false",
"disk_discard": "unmap",
"disk_image": "false",
"disk_size": "20480",
"existing_ansible_ssh_args": "{{env ANSIBLE&#95;SSH&#95;ARGS}}",
"firmware": "",
"format": "qcow2",
"headless": "true",
"http_directory": "./packer/qemu/linux/{{user distro&#95;name}}/http/",
"kubernetes_cni_deb_version": null,
"kubernetes_cni_http_source": null,
"kubernetes_cni_semver": null,
"kubernetes_cni_source_type": null,
"kubernetes_container_registry": null,
"kubernetes_deb_gpg_key": null,
"kubernetes_deb_repo": null,
"kubernetes_deb_version": null,
"kubernetes_http_source": null,
"kubernetes_load_additional_imgs": null,
"kubernetes_rpm_gpg_check": null,
"kubernetes_rpm_gpg_key": null,
"kubernetes_rpm_repo": null,
"kubernetes_rpm_version": null,
"kubernetes_semver": null,
"kubernetes_series": null,
"kubernetes_source_type": null,
"machine_id_mode": "444",
"memory": "2048",
"oem_id": "",
"output_directory": "./output/{{user build&#95;name}}-kube-{{user kubernetes&#95;semver}}",
"python_path": "",
"qemu_binary": "qemu-system-x86_64",
"ssh_password": "$SSH_PASSWORD",
"ssh_username": "builder",
"vm_name": "{{user build&#95;name}}-kube-{{user kubernetes&#95;semver}}",
"vnc_bind_address": "127.0.0.1"
}
}
Debugging Steps Taken
  1. Increased timeouts/retries: No change.

  2. Verified credentials: Manual SSH works post-failure (if I pause the build).

  3. Added verbose logging: Packer debug logs show the VM boots, but SSH dies abruptly.

  4. Tested locally and in CI: Same error on both.

Suspected Causes
  1. SSH service instability: The VM might not finish SSH setup before Packer connects.

  2. Cloud-init/autoinstall misconfiguration: Ubuntu 24.04’s autoinstall could be skipping critical steps.

  3. QEMU networking: NAT/port forwarding might interfere.

Questions for the Community
  1. Has anyone hit this with Ubuntu 24.04’s autoinstall?

  2. Are there known workarounds for Packer + QEMU SSH flakes?

  3. How can I inspect the VM’s SSH service mid-build? (e.g., console logs?)

Additional Context
  • Packer boot command:

{
"boot_command_prefix": "clinux /casper/vmlinuz --- autoinstall ds='nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/24.04/'initrd /casper/initrdboot",
"build_name": "ubuntu-2404",
"distribution_version": "2404",
"distro_name": "ubuntu",
"guest_os_type": "ubuntu-64",
"iso_checksum": "d6dab0c3a657988501b4bd76f1297c053df710e06e0c3aece60dead24f270b4d",
"iso_checksum_type": "sha256",
"iso_url": "",
"os_display_name": "Ubuntu 24.04",
"shutdown_command": "shutdown -P now",
"unmount_iso": "true"
}
Any tips would be greatly appreciated! I’m happy to provide more logs or test suggestions.

Karine Santos
2025-05-21 17:03:07
/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/.local/bin/packer build -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/kubernetes.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/cni.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/containerd.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/wasm-shims.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/ansible-args.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/goss-args.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/common.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/additional_components.json"  -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/config/ecr_credential_provider.json"  -color=true -var-file="/builds/magalu-cloud-iaas/k8s/image-builder/images/capi/packer/qemu/qemu-ubuntu-2404.json"  packer/qemu/packer.json
8401qemu: output will be in this color.
8402==> qemu: Retrieving ISO
8403==> qemu: Trying
8404==> qemu: Trying
8405==> qemu: Download failed context deadline exceeded
8406==> qemu: error downloading ISO: [context deadline exceeded]
8407Build 'qemu' errored after 30 minutes 515 milliseconds: error downloading ISO: [context deadline exceeded]
8408==> Wait completed after 30 minutes 515 milliseconds
8409==> Some builds didn't complete successfully and had errors:
8410--> qemu: error downloading ISO: [context deadline exceeded]
8411==> Builds finished but no artifacts were created.
8412make[2]: ****** [Makefile:560: build-qemu-ubuntu-2404] Error 1
8413make[2]: Leaving directory '/builds/magalu-cloud-iaas/k8s/image-builder/images/capi'
8414make[1]: ****** [Makefile:1245: mgc-build-image] Error 2
8415make[1]: Leaving directory '/builds/magalu-cloud-iaas/k8s/image-builder/images/capi'
8416make: ****** [Makefile:37: mgc-build-image] Error 2

8417
Cleaning up project directory and file based variables
00:01
8418ERROR: Job failed: command terminated with exit code 1
Victor Sartori
2025-05-21 19:22:02

@Karine Santos Looking at the messages, this error seems related to the ISO download.
The first error indicates a timeout while downloading the ISO file.

👍 Sriraman Srinivasan
eszanon
2025-05-26 14:06:54

@eszanon has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2025-05-27 16:22:30

Image-builder v0.1.44 is now available:
Thanks to all contributors! 💙

GitHub
🎺 mboersma
:cluster_api: mboersma
:kubernetes_party: mboersma, Drew Hudson-Viles
Michael Nielson
2025-05-30 22:26:17

@Michael Nielson has joined the channel

Slackbot
2025-06-02 15:30:15

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-02 15:32:33

☝️ This is the updated agenda doc -
(@mboersma when you have time would you mind recreating that reminder with the new link?)

👍 mboersma
Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-02 15:38:56
mboersma
2025-06-02 16:04:06

set up a reminder “https://docs.google.com/document/d/100uv2GmlgWyLBVP65W6ABNJ_EqbvVYTYtTilCLbnVYI/edit” in this channel at 8:30AM every other Monday (next occurrence is June 16th), Mountain Daylight Time.

:thank_you_icon: Marcus Noble
mboersma
2025-06-02 16:05:46

set up a reminder “Image-Builder office hours start in 1 hour. Agenda: https://docs.google.com/document/d/100uv2GmlgWyLBVP65W6ABNJ_EqbvVYTYtTilCLbnVYI/edit” in this channel at 8:30AM every other Monday (next occurrence is June 16th), Mountain Daylight Time.

runzhliu
2025-06-04 16:26:58

@runzhliu has joined the channel

Sriraman Srinivasan
2025-06-09 08:17:08

@Marcus Noble / @mboersma PR needs your approvals for test and merger. The changeset primarily targets vSphere OVA builds.
cc: @palnabarun

GitHub
Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-09 08:25:22

It's on my list. Will try and get to it today but so far my day isn't going too well 😅

🙏 Sriraman Srinivasan
Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-09 15:35:48

Is there any more context to this? There's no related issues nor any explanation why it's needed or what the impact would be.

Sriraman Srinivasan
2025-06-10 04:39:58

Will create a feature request with the details and also update the PR with the context. In nutshell, this more wrt to improving the node performance.

palnabarun
2025-06-11 11:16:46

@Marcus Noble added some context to . Thanks again for taking a look!

🙏 Sriraman Srinivasan
Sriraman Srinivasan
2025-06-11 13:03:30

@Marcus Noble Please do let me know if you need any further information.

Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-12 08:41:55

👍 lgtm

Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-12 08:42:03

Thank you for the context! 💙

chrischdi
2025-06-10 09:52:21

Hey folks, can I get a review for ? This re-adds the presubmit vsphere/OVA CI job (as optional) again, based on the new community based infra.

I’m doing the testing and required changes to make it finally work in

GitHub
💙 Marcus Noble, rajas
:git_merge: dims
chrischdi
2025-06-11 09:49:05

Note: the image-builder PR would now be open for a review 🙂 the presubmit is green on that PR

Yuanliang Zhang
2025-06-10 18:13:27

@Yuanliang Zhang has joined the channel

Yuanliang Zhang
2025-06-10 18:15:43

Anybody aware of any known PR check-in failure /test pull-gcp-all? This is not related with my change Make kubelet starting as a windows service by zylxjtu · Pull Request #1752 · kubernetes-sigs/image-builder, so I'm wondering if there are any known (infra-related maybe?) issue? Thanks!

GitHub
palnabarun
2025-06-11 11:16:15

@palnabarun has joined the channel

Rémy Léone
2025-06-13 17:23:18

@Rémy Léone has joined the channel

Rémy Léone
2025-06-13 17:25:21

@Marcus Noble Hello 👋 I'm working with @Tomy Guichard at Scaleway with @Leïla MARABESE We can have a conversation about what you need for integrating image-builder with Scaleway 🙂

:gr_blob_wave: Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-16 07:21:37

👋 Hey y'all!

I just read the comment on the PR 🙂 Really happy to see Scaleway is willing to at least contribute effort to supporting this new provider! 💙 From my perspective having y'all down as reviewers for this provider would be enough (for now) for me to be happy to add this provider to image-builder. (@mboersma @Drew Hudson-Viles @jsturtevant do y'all agree? 🙂)

If possible I would LOVE to see some infra support from Scaleway to support testing of PR as we're very lacking in that regard across image-builder. It would go a long way to making Scaleway a solid, reliable option in image-builder but I also know how difficult these kind of agreements can be so I'm staying realistic 🙂

On a side note - very happy to see the Scaleway provider officially supported 😄 I'm a Scaleway user myself although just a single cluster plus some other resources.

GitHub
🎉 Sriraman Srinivasan, rajas
Sriraman Srinivasan
2025-06-16 08:14:42

Welcome!!! 🎉

Drew Hudson-Viles
2025-06-16 08:21:15

Yo! Just here to echo Marcus really. Anything that can be provided be it in terms of supporting the provider and, if possible of course, infra for testing the provider would be amazing. Welcome!

:parrotk8s: Marcus Noble
Rémy Léone
2025-06-16 09:35:02

On a credentials sharing perspectives, what are the process on the CNCF side to share those kind of access? Are they organic and scope based, or is there within the CNCF a way for vendor to share this kind of resources ?

Rémy Léone
2025-06-16 09:35:27

Scaleway organization supports projects and we can have several projects within an organization

Rémy Léone
2025-06-16 09:35:33

We have support for IAM permissions

Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-16 09:36:19

I would need to check with CNCF. I'm not totally sure myself. Maybe @mboersma knows more but he's currently on vacation.

Rémy Léone
2025-06-16 09:36:20

So I'm trying to envision what would be the best architecture for it, what would be the email address to invite in the organization, how to track credentials and have audit on who can access what

Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-16 09:37:22

Maybe @bentheelder could share some insight on how providers go about donating cloud resources for testing 🙏

Sriraman Srinivasan
2025-06-16 09:39:06

@chrischdi may be also be able to provide some insight here?

chrischdi
2025-06-16 09:46:11

From my experience it is not only about sharing credentials. The infrastructure must be owned by the CNCF/Community if I know it right. So its more about donating credits or money so the community could leverage that. But I’m not the one who went through the process. Maybe best to ask in test-infra folks on what viable variants are.

chrischdi
2025-06-16 09:47:19

But in our case (vSphere) it was different, because we still use a public cloud (in our case Google Cloud) for the infrastructure, and are not a cloud provider ourselves.

bentheelder
2025-06-16 17:31:56

Please see this thread:

michael mccune (https://kubernetes.slack.com/team/U11HJAX1S)
💙 Marcus Noble, Sriraman Srinivasan
:thank_you_icon: Marcus Noble
👍 Sriraman Srinivasan
Tomy Guichard
2025-06-13 17:25:25

@Tomy Guichard has joined the channel

Leïla MARABESE
2025-06-13 17:25:25

@Leïla MARABESE has joined the channel

Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-16 10:25:42

⚠️ Looks like this Slack workspace if moving to the free tier this week 😳 Which if I understand correctly means we'll lose the history older than 90 days in this channel.

If there is anything from the history in this channel that you refer to please mention it in this thread and I'll try and get it backed up somewhere more permanent (GitHub or similar).

Changes to Kubernetes Slack | Kubernetes Contributors

Kubernetes Contributors
👀 Drew Hudson-Viles
Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-16 10:40:36

We're seeing some GCP failures in our PR tests, does anyone know / can confirm for me if the official Ubuntu 20.04 base image is no longer available on GCP now that support has ended?

googlecompute.ubuntu-2004: Error getting source image for instance creation: Could not find image, ubuntu-2004-lts, in projects

Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-16 11:03:33

Unless anyone shouts at me not to, I propose removing Ubuntu 20.04 from GCP as it's EOL anyway.

GitHub
👍 mboersma
Slackbot
2025-06-16 15:30:06

Reminder: Image-Builder office hours start in 1 hour. Agenda:

Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-16 15:32:58

I’m not able to make it today but I have added some points to the agenda

mboersma
2025-06-16 15:55:34

I'll be there, hopefully others can join.

💙 Marcus Noble
:thx_thanks: Marcus Noble
Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-16 16:31:51

I might actually be able to going from my phone in about 5 min

Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-16 17:09:04

Slight correction to what I said @mboersma - the public channels are archived but they’re offline and not currently searchable so we can’t reference them. (https://github.com/kubernetes/community/blob/master/communication/slack-migration-faq.md#what-information-will-we-lose)

GitHub
Drew Hudson-Viles
2025-06-16 17:53:28

Sorry, my bad. Had to set off to pick Ada up from nursery early today and forgot to drop a message in here. Today has flown by so time got away with me and before I knew it I was heading out and already this is done 🤦‍♂️🤦‍♂️

Marcus Noble (k8s@marcusnoble.co.uk)
2025-06-16 17:56:24

No worries. Been a wild day for me too.

bentheelder
2025-06-16 17:31:46

@bentheelder has joined the channel